Original: C ONSTANT C URVATURE G RAPH C ONVOLUTIONAL N ETWORKS
Revised: C ONSTANT C URVATURE G RAPH C ONVOLUTIONAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Interest has been rising lately towards methods representing data in non-Euclidean spaces, e.g. hyperbolic or spherical. These geometries provide speciﬁc inductive biases useful for certain real-world data properties, e.g. scale-free or hierarchical graphs are best embedded in a hyperbolic space.
Revised: Interest has been rising lately towards methods representing data in non-Euclidean spaces, e.g. hyperbolic or spherical, that provide speciﬁc inductive biases useful for certain real-world data properties, e.g. scale-free, hierarchical or cyclical.

Original: However, thevery popularclass of graph neural networks is currently limited to model data only via Euclidean node embeddings and associated vector space operations.
Revised: However, the popular graph neural networks are currently limited in modeling data only via Euclidean geometry and associated vector space operations.

Original: In this work, we bridge this gap by proposing mathematically grounded generalizations of graph convolutional networks (GCN) to (products of) constant curvature spaces.
Revised: Here, we bridge this gap by proposing mathematically grounded generalizations of graph convolutional networks (GCN) to (products of) constant curvature spaces.

Original: We do this by i) extending the gyro-vector space theory from hyperbolic to spherical spaces, providing a uniﬁed and smooth view of the two geometries, ii) leveraging gyrobarycentric coordinates that generalize the classic Euclidean concept of the center of mass .
Revised: We do this by i) introducing a uniﬁed formalism that can interpolate smoothly between all geometries of constant curvature, ii) leveraging gyro-barycentric coordinates that generalize the classic Euclidean concept of the center of mass .

Original: Our class of models gives strict generalizations in the sense that they recover their Euclidean counterparts when the curvature goes to zero from either side.
Revised: Our class of models smoothly recover their Euclidean counterparts when the curvature goes to zero from either side.

Original: Empirically, our methods outperformdifferent types of classic Euclidean GCNs in the tasks of node classiﬁcation andminimizing distortion  for symbolic data exhibiting non-Euclidean behavior, according to their discrete curvature.
Revised: Empirically, we outperform Euclidean GCNs in the tasks of node classiﬁcation and distortion minimization for symbolic data exhibiting non-Euclidean behavior, according to their discrete curvature.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Revised: Graph Convolutional Networks.

Revised: The success of convolutional networks and deep learning for image data has inspired generalizations for graphs for which sharing parameters is consistent with the graph geometry.

Revised: Bruna et al.

Revised: ; Henaff et al.

Original: Euclidean geometry in ML.
Revised: (2015) are the pioneers of spectral graph convolutional neural networks in the graph Fourier space using localized spectral ﬁlters on graphs. However, in order to reduce the graph-dependency on the Laplacian eigenmodes, Defferrard et al. (2016) approximate the convolutional ﬁlters using Chebyshev polynomials leveraging a result of Hammond et al. The resulting method (discussed in appendix A) is computationally efﬁcient and superior in terms of accuracy and complexity.

Revised: Further, Kipf & Welling (2017) simplify this approach by considering ﬁrst-order polynomials approximations obtaining high scalability.

Revised: The proposed graph convolutional networks (GCN) is interpolating node embeddings via a symmetrically normalized adjacency matrix, while this weight sharing can be understood as an efﬁcient diffusion-like regularizer.

Revised: Recent works extend GCNs to achieve state of the art results for link prediction (Zhang & Chen, 2018), graph classiﬁcation (Hamilton et al., 2017; Xu et al., 2018) and node classiﬁcation (Klicpera et al., 2019; Veliˇckovi´c et al., 2018).

Original:  In machine learning (ML), data is most often represented in a Euclidean space for various reasons.
Revised: Euclidean geometry in ML. In machine learning (ML), data is most often represented in a Euclidean space for various reasons.

Original: First, some data is intrinsically Euclidean, such as positions in 3D space in classical mechanics.
Revised: First, some data is intrinsically Euclidean, such as positions in 3D space in classical mechanics.

Original: Second, intuition is easier in such spaces, as they possess an appealing vectorial structure allowing basic arithmetic and a rich theory of linear algebra.
Revised: Second, intuition is easier in such spaces, as they possess an appealing vectorial structure allowing basic arithmetic and a rich theory of linear algebra.

Original: Finally, a lot of quantities of interest such as distances and inner-products are known in closed-form formulae and can be computed very efﬁciently on the existing hardware.
Revised: Finally, a lot of quantities of interest such as distances and inner-products are known in closed-form formulae and can be computed very efﬁciently on the existing hardware.

Original: These operations are the basic building blocks for most of today’s popular machine learning models.
Revised: These operations are the basic building blocks for most of today’s popular machine learning models.

Original: Thus, the powerful simplicity and efﬁciency of
Revised: Thus, the powerful simplicity and efﬁciency of

Original: Euclidean geometry has led to numerous methods achieving state-of-the-art on tasks as diverse as machine translation Bahdanau et al. (2014); Vaswani et al. speech recognition Graves et al.
Revised: Euclidean geometry has led to numerous methods achieving state-of-the-art on tasks as diverse as machine translation (Bahdanau et al., 2014; Vaswani et al., 2017), speech recognition (Graves et al., 2013), image classiﬁcation (

Original: (2013), image classiﬁcation He et al. (2016) or recommender systems He et al.
Revised: He et al., 2016) or recommender systems (He et al., 2017).

Original: Riemannian ML.
Revised: Riemannian ML.

Original: In spite of this success, certain types of data  have been shown to be better represented by other types of geometries 
Revised: In spite of this success, certain types of data (e.g. hierarchical, scale-free or spherical data) have been shown to be better represented by non-Euclidean geometries (Defferrard et al., 2019;

Original: Bronstein et al. ; Nickel & Kiela (2017), leading in particular to the rich theories of manifold learning Roweis & Saul (2000); Tenenbaum et al. and information geometry Amari & Nagaoka (2007).
Revised: Bronstein et al., 2017; Nickel & Kiela, 2017; Gu et al., 2019), leading in particular to the rich theories of manifold learning (Roweis & Saul, 2000; Tenenbaum et al., 2000) and information geometry (Amari & Nagaoka, 2007).

Original: The mathematical framework in vigor to manipulate non-Euclidean geometries is known as Riemannian geometry Spivak (1979).
Revised: The mathematical framework in vigor to manipulate nonEuclidean geometries is known as Riemannian geometry (Spivak, 1979).

Original: We brieﬂy discuss some concepts of this machinery in appendix B. Although its theory leads to many strong and elegant results, some of its basic quantities such as the distance function d ( · , · ) are in general not available in closed-form, which can be prohibitive to many computed-based methods.
Revised: Although its theory leads to many strong and elegant results, some of its basic quantities such as the distance function d ( · , · ) are in general not available in closed-form, which can be prohibitive to many computational methods.

Original:  Geometries of Constant Curvature.
Revised: Representational Advantages of Geometries of Constant Curvature.

Original: An interesting trade-off between general Riemannian manifolds and the Euclidean space is given by manifolds of constant sectional curvature .
Revised: An interesting trade-off between general Riemannian manifolds and the Euclidean space is given by manifolds of constant sectional curvature .

Original: They deﬁne together what are called hyperbolicgeometry (negative curvature), ellipticgeometry (positive curvature) and Euclideangeometry (zero curvature).
Revised: They deﬁne together what are called hyperbolic (negative curvature), elliptic (positive curvature) and Euclidean (zero curvature) geometries.

Original: One of their advantage is that most of the quantities of interest for machine learning methods are known in closed-form formulae: distance, geodesics, exponential map, parallel transport, as well as their gradients.
Revised: As discussed below and in appendix B,

Original: These geometries also possess their quickly "runs out of space" when ﬁtting exponentially volume growing data such as trees.
Revised: Euclidean spaces have limitations and suffer from large distortion when embedding certain types of data such as trees, e.g. ﬁg. 1.

Original: This issue is formalised in appendix C.
Revised: In these cases, the hyperbolic and spherical spaces have representational advantages providing a better inductive bias for the respective data.

Original: The hyperbolic space can be intuitively understood as a continuous tree: the volume of a ball grows exponentially with its radius, similarly as how the number of nodes in a binary tree grows exponentially with its depth.
Revised: The hyperbolic space can be intuitively understood as a continuous tree: the volume of a ball grows exponentially with its radius, similarly as how the number of nodes in a binary tree grows exponentially with its depth.

Original: Its tree-likeness properties have long been studied mathematically Gromov (1987); Hamann (2017); Ungar (2008), were related to heterogeneous properties of complex networks Krioukov et al. and used to visualize large hierarchies Lamping et al. Recently, in machine learning, hyperbolic representations outperformed their Euclidean analogues in a number of embedding and classiﬁcation tasks Cho et al.
Revised: Its tree-likeness properties have long been studied mathematically (Gromov, 1987; Hamann, 2017; Ungar, 2008) and it was proven to better embed complex networks (Krioukov et al., 2010), scale-free graphs and hierarchical data compared to the Euclidean geometry (Cho et al., 2019;

Original: De Sa et al.

Original: ; Ganea et al. (2018a); Gu et al. ; Nickel & Kiela (2018; 2017); Tifrea et al.
Revised: Sala et al., 2018; Ganea et al., 2018b; Gu et al., 2019; Nickel & Kiela, 2018; 2017; Tifrea et al., 2019).

Original: Several important tools or methods found their hyperbolic counterparts, such as variational autoencoders Mathieu et al.
Revised: Several important tools or methods found their hyperbolic counterparts, such as variational autoencoders (Mathieu et al., 2019;

Original: ; Ovinnikov (2019), attention mechanisms Gulcehre et al. (2018), matrix multiplications, recurrent units and multinomial logistic regression Ganea et al. (2018b).
Revised: Ovinnikov, 2019), attention mechanisms (Gulcehre et al., 2018), matrix multiplications, recurrent units and multinomial logistic regression (Ganea et al., 2018a).

Original: Similarly, spherical geometry provides beneﬁts for modeling certain types of data Matousek (2013); Davidson et al.
Revised: Similarly, spherical geometry provides beneﬁts for modeling spherical or cyclical data (Defferrard et al., 2019;

Original: (2018); Xu & Durrett (2018); Gu et al. ; Grattarola et al. ; Wilson et al.
Revised: Matousek, 2013; Davidson et al., 2018; Xu & Durrett, 2018; Gu et al., 2019; Grattarola et al., 2018; Wilson et al., 2014).

Original: Graph Embeddings. The Euclidean space has certain limitations of embedding various types of graphs without large distortion.
Revised: Computational Efﬁciency of Constant Curvature Spaces (CCS). CCS are some of the few

Original: In some of these cases (e.g. scale-free graphs and complex networks Krioukov et al. (2010)), the hyperbolic and spherical spaces do not suffer from this representational bottleneck and can provide a better inductive bias for the speciﬁc data types.
Revised: Riemannian manifolds to possess closed-form formulae for geometric quantities of interest in computational methods, i.e. distance, geodesics, exponential map, parallel transport and their gradients.

Original: We detail this discussion in appendix C.
Revised: We also leverage here the closed expressions for weighted centroids.

Original: "Linear Algebra" of Constant Curvature Spaces – Gyrovector Spaces.
Revised: “Linear Algebra” of CCS: Gyrovector Spaces.

Original: The hyperbolic space is especially useful in Einstein’s special relativity theory where particles’ velocities are mathematically represented as vectors in the Poincaré ball model.

Original: In order to study this geometry  in analogy with thevector space formalism of the Euclidean geometry, Ungar (1999; 2005; 2008; 2016) proposed the elegant non-associative algebraic formalism of gyrovector spaces1 . Relativistic composition of velocities now becomes the gryo-addition .
Revised: In order to study the geometry of constant negative curvature in analogy with the Euclidean geometry, Ungar (1999; 2005; 2008; 2016) proposed the elegant non-associative algebraic formalism of gyrovector spaces .

Original: Recently, Ganea et al.
Revised: Recently, Ganea et al.

Original: (2018b) have generalized matrix multiplication and pointwise non-linearities to gyro spaces, allowing them to stack these building blocks for generalizing classic deep learning models to operate with hyperbolic data representations.
Revised: (2018a) have linked this framework to the Riemannian geometry of the space, also generalizing the building blocks for non-Euclidean deep learning models operating with hyperbolic data representations.

Original: However, it remains unclear how to extend in a principled manner the  gyrovector space theory to spaces of constant positive curvature (spherical) .
Revised: However, it remains unclear how to extend in a principled manner the connection between Riemannian geometry and gyrovector space operations for spaces of constant positive curvature (spherical) .

Original: By leveraging Euler’s formula and complex analysis,as far as we know, we present  the ﬁrst uniﬁed gyro framework that smoothly interpolates between geometries of constant curvatures irrespective of their signs.Tools especially important for machine learning such as exponential map, geodesics, parallel transport, distance, barycentric coordinates have efﬁcient closed form expressions in these spaces and converge to their Euclidean variants when the curvature vanishes from both positive and negative regimes. This further allows us to develop deep learning architectures in products of constant curvature spaces. Graph Neural Networks.
Revised: By leveraging Euler’s formula and complex analysis, we present to our knowledge the ﬁrst uniﬁed gyro framework that smoothly interpolates between geometries of constant curvatures irrespective of their signs.

Original: Kipf & Welling (2016) recently proposed a scalable approach to train deep network models directly on graphs, sharing parameters in a manner that is consistent with the geometry of the graph. As the weight sharing scheme is inspired from graph convolutions, involving the graph Laplacian, they were coined graph convolutional networks (GCN). The proposed architecture is essentially made of interpolating node embeddings via a (symmetrically) normalized adjacency matrix, while this weight sharing can be understood as an efﬁcient diffusion-like regularizer. Similar works Abu-El-Haija et al.
Revised: This is possible when working with the Poincaré ball and stereographic spherical projection models of respectively hyperbolic and spherical spaces.

Original: Hamilton et al. (2017); Defferrard et al. ; Klicpera et al. Veliˇckovi´c et al.
Revised: How should one adapt graph neural networks to non-ﬂat geometries of constant curvature?

Original: (2018) showed success at a number of tasks including link prediction

Original: Zhang & Chen (2018), graph classiﬁcation Xu et al. and node classiﬁcation Kipf & Welling (2017); Klicpera et al. Veliˇckovi´c et al. We describe GCN and discuss some related work in more detail in appendix A, but a broader overview is given by Wu et al.
Revised: In this work, we propose constant curvature GCNs to model non-Euclidean data.

Original: How should one adapt graph neural networks to non-ﬂat geometries of constant curvature? In this work, we propose to let GCN manipulate data via node embeddings lying in spaces of constant curvature or product of those, instead of a Euclidean space, in order to combine the representational power of these geometries with the effectiveness of GCNs.
Revised: Node embeddings lie in spaces of constant curvature or product of those instead of a Euclidean space, thus leveraging both the representational power of these geometries and the effectiveness of GCNs.

Original: Our contributions are as follows :
Revised: Concurrent to our work, Chami et al.

Original: • We extend the gyro-vector space theory from hyperbolic to spherical spaces using complex analysis. This gives uniﬁed and computationally efﬁcient expressions for important Riemannian geometric tools needed in machine learning, e.g. matrix multiplication and barycentric coordinates. Our models are smooth deformations of the Euclidean vector space machinery. • We leverage this apparatus to develop mathematically grounded adaptations of GCN to (products of) spaces of constant curvature.
Revised: Liu et al.

Original: • We empirically show the beneﬁt of our models on several real and synthetic node classiﬁcation and distortion minimization benchmarks.
Revised: (2019) propose hyperbolic graph neural networks using tangent space aggregation.

Original:  T HE G EOMETRY OF C ONSTANT C URVATURE S PACES
Revised:  T HE G EOMETRY OF C ONSTANT C URVATURE S PACES

Original: Riemannian Geometry.
Revised: Riemannian Geometry.

Original: A manifold M of dimension d is a generalization to higher dimensions of the notion of surface, and is a space that locally looks like R d .
Revised: A manifold M of dimension d is a generalization to higher dimensions of the notion of surface, and is a space that locally looks like R d .

Original: At each point x ∈ M , M can be associated a tangent space T x M , which is a vector space of dimension d that can be understood as a ﬁrst order approximation of M around x .
Revised: At each point x ∈ M , M can be associated a tangent space T x M , which is a vector space of dimension d that can be understood as a ﬁrst order approximation of M around x .

Original: A riemannian metric g is given by an inner-product g x ( · , · ) at each tangent space T x M , g x varying smoothly with x .
Revised: A riemannian metric g is given by an inner-product g x ( · , · ) at each tangent space T x M , g x varying smoothly with x .

Original: A given g deﬁnes the geometry of M , because it can be used to deﬁne the distance between x and y as the inﬁmum of the lengths of smooth paths γ : [0 , 1] → M from x to y , where the length is deﬁned as (cid:96) ( γ ) :
Revised: A given g deﬁnes the geometry of M , because it can be used to deﬁne the distance between x and y as the inﬁmum of the lengths of smooth paths γ : [0 , 1] → M from x to y , where the length is deﬁned as (cid:96) ( γ ) :

Original: = (cid:82) 1 0 (cid:112) g γ ( t ) (˙ γ ( t ) , ˙ γ ( t ))
Revised: = (cid:82) 1 0 (cid:112) g γ ( t ) (˙ γ ( t ) , ˙ γ ( t ))

Original: d t .
Revised: d t .

Original: Under certain assumptions, a given g also deﬁnes  curvature . We detail these concepts in appendix B.
Revised: Under certain assumptions, a given g also deﬁnes a curvature at each point.

Original: The Poincaré model.
Revised: Unifying all curvatures κ . There exist several models of respectively constant positive and negative curvatures.

Original: We chooseto work in the Poincaré model of the hyperbolic geometry because it can be continuously deformed to the Euclidean space. For a curvature − c < 0 , the Poincaré model of dimension d ≥ 2 is deﬁned as P dc := { x ∈ R d | c (cid:107) x (cid:107) 2 < 1 } equipped with its Riemannian metric g P x = 4 (1 − c
Revised: For positive curvature, we choose the stereographic projection of the sphere, while for negative curvature we choose the Poincaré model which is the stereographic projection of the Lorentz model.

Original: ||
Revised: As explained below, this choice allows us to generalize the gyrovector space framework and unify spaces of both positive and negative curvature κ into a single model which we call the κ -stereographic model. The κ -stereographic model.

Original: [-x || 2 ) 
 I = ( λ cx )-] 2 I . What is convenient however, is that the distance induced by g between x, y ∈ P dc is given by the formula:
Revised: [-x || 2 ) 
 I = ( λ cx )-]For a curvature κ ∈ R and a dimension d ≥ 2 , it is deﬁned as st dκ = { x ∈ R d | − κ (cid:107) x (cid:107) 22 < 1 } equipped with its Riemannian metric g κ x =

Original: [Equation]
Revised: [Equation]

Original: Notice that here it holds that d c P ( x, y ) c −→ 0−−−→ 2 || x − y || , so in the limit we recover Euclidean geometry as desired.
Revised: Note in particular that when κ ≥ 0 , st dκ is R d , while when κ < 0 it is the open ball of radius 1 / √− κ .

Original: Gyrovector spaces and Riemannian geometry of the Poincaré ball.
Revised: Gyrovector spaces & Riemannian geometry.

Original: As discussed in section 1, the gryovector space formalism is used to generalize vector spaces to  hyperbolic geometry.
Revised: As discussed in section 1, the gyrovector space formalism is used to generalize vector spaces to the Poincaré model of hyperbolic geometry (Ungar, 2005; 2008).

Original: In addition, important quantities from Riemannian geometry can be rewritten in terms of the Möbius vector addition and scalar-vector multiplication Ganea et al. (2018b).
Revised: In addition, important quantities from Riemannian geometry can be rewritten in terms of the Möbius vector addition and scalar-vector multiplication (Ganea et al., 2018a).

Original: We deﬁne and detail gyrovector spaces in appendix D. For x, y ∈ P dc ,  the Möbius addition is given by:
Revised: We here extend gyrovector spaces to the κ -stereographic model, i.e. allowing positive curvature. For κ > 0 and any point x ∈ st dκ , we will denote by ˜ x the unique point of the sphere of radius κ −in R d +1 whose stereographic projection is x . As detailed in appendix C.2.2, it is given by

Original: [Equation]
Revised: [Equation]

Original: Fors ∈ R and x  ∈ ⊗ c M ,  the scalar multiplication in the Poincaré modelP dc is given by:
Revised: For x , y ∈ st dκ , we deﬁne the κ - addition , in the κ -stereographic model by:

Original: [Equation]
Revised: [Equation]

Original: One can show that Möbius addition and scalar multiplication in P dc form a gyrovector space Ungar (2008) which is detailed in appendix D. We can use the Möbius addition to rewrite the distance formula as d c P ( x, y  ) 
Revised: The κ -addition is deﬁned in all the cases except for spherical geometry and x = y / ( κ (cid:107) y (cid:107) 2 ) as stated by the following theorem proved in Appendix C.2.1. Theorem 1 (Deﬁniteness of κ -addition) .

Original: = 2 √ c tanh − 1( √ c || −  x ⊕ c M y  ||) .
Revised: We have 1 − 2 κ x T y + κ 2 ||

Original: This model is smoothly deformed to the
Revised: x || 2 ||

Original: Euclidean model in the curvature limit as d c P ( x, y ) c −→ 0 −−−→ 2 (cid:107){+= 
 if and only if κ > 0 and+} x − y  (cid:107)  .
Revised: y || 2 {+= 
 if and only if κ > 0 and+} x = y / ( κ (cid:107) y (cid:107) 2 ) .

Original: For x, y ∈P dc , the geodesic γ x −→ y : R−→ P dc , connecting x and y is given by
Revised: For s ∈ R and x ∈ st d κ (and | s tan − 1 κ (cid:107) x (cid:107)| < κ 12 π/ 2 if κ > 0 ), the κ -scaling in the κ -stereographic model is given by:

Original: [Equation]
Revised: [Equation]

Original: Finally, the exponential and logarithmic map are written in closed form Ganea et al.
Revised: 3) where tan κ equals κ − 1 / 2 tan if κ > 0 and ( − κ ) − 1 / 2 tanh if κ < 0 . This formalism yields simple closed-forms for various quantities including the distance function inherited from the Riemannian manifold ( st dκ , g κ ), the exp and log maps, and geodesics, as shown by the following theorem.

Original: (2018b):
Revised: Theorem 2 (Extending gyrovector spaces to positive curvature) .

Original: For x, y ∈ P dc and v (cid:54) =0 , y  (cid:54) = x , the exponential map exp x : T x P dc −→ P dc and the log map log x : P dc −→ T x P dc are given by 
Revised: For x , y ∈ st dκ , x (cid:54) = y , v (cid:54) = 0 , (and x (cid:54) = − y / ( κ (cid:107) y (cid:107) 2 ) if κ > 0 ), the distance function is given by a :

Original: [Equation]
Revised: [Equation]

Original: Gyrovector spaces of the Spherical Spaces. The gyrovector space formalism was so far, to our knowledge, only described for hyperbolic spaces.
Revised: the unit-speed geodesic from x to y is unique and given by

Original: So, how can we extend it to spaces of constant positive curvature (spherical) K ?
Revised: [Equation] and ﬁnally the exponential and logarithmic maps are described as: [Equation]

Original: It turns out that replacing c by − K in  theabove equations of the Poincaré model sufﬁces.
Revised: Proof sketch: The case κ ≤ 0 was already taken care of by (Ganea et al., 2018a). For κ > 0 , we provide a detailed proof in Appendix C.2.2. The exponential map and unit-speed geodesics are obtained using the

Original: However, that implies that all occurrences of √ c would be replaced by i √ K where i is theimaginary unit. This will still give only real functions if one uses Euler’s formula e ix = cos( x  ) + i sin( x  ) and the resulting identities tan( ix )
Revised: Egregium theorem and the known formulas in the standard spherical model. The distance then follows from the formula d κ ( x , y ) = (cid:107) log κ x ( y ) (cid:107) x which holds in any Riemannian manifold.

Original:  = 
Revised: (cid:3) Around κ = 0 . One notably observes that choosing κ = 0 yields all corresponding Euclidean quantities, which guarantees a continuous interpolation between κ -stereographic models of different curvatures, via Euler’s formula tan

Original: i tanh( x ) and tan − 1 ( ix ) 
Revised: ( x ) = − i tanh( ix ) where i :=

Original: =
Revised: √− 1 . But is this interpolation differentiable with respect to κ ? It is as shown by the following theorem, proved in Appendix C.2.3. Theorem 3 (Smoothness of st dκ w.r.t.

Original: i tanh − 1 ( x ) .
Revised: κ around 0 ) .

Original: In this case, all the above equations will recover the corresponding quantities from the space of constant curvature obtained by isometrically projecting the sphere S dc into R d , i.e. the stereographic projection of the sphere.
Revised: Let v (cid:54) = 0 and x , y ∈ R d , such that x (cid:54) = y (and x (cid:54) = − y / ( κ (cid:107) y (cid:107) 2 ) if κ > 0 ). Quantities in Eqs.

Original: Moreover, the gyrovector space deﬁnition list would be fully satisﬁed as well.{+(4,5,6) are well-deﬁned for | κ | < 1 / min( (cid:107) x (cid:107) 2 , (cid:107) y (cid:107) 2 ) , i.e. for κ small enough. Their ﬁrst order derivatives at 0 − and 
+}
Revised: {+(4,5,6) are well-deﬁned for | κ | < 1 / min( (cid:107) x (cid:107) 2 , (cid:107) y (cid:107) 2 ) , i.e. for κ small enough. Their ﬁrst order derivatives at 0 − and 
+}

Original: We denote this gyrospace and  the two resulting Möbius gyro operations by ( G = R d , ⊕ c G , ⊗ c G ) .
Revised: + exist and are equal. Moreover, for the distance we have: [Equation]

Original: We discuss this in detail in appendix D.
Revised: [Equation] Note that for x T y ≥ 0 , this tells us that an inﬁnitesimal change of curvature from zero to small negative, i.e. towards 0 − , while keeping x , y ﬁxed, has the effect of increasing their distance.

Original: As a consequence, we have a uniﬁed formalism that interpolates smoothly between all three geometries of constant curvature.
Revised: As a consequence, we have a uniﬁed formalism that interpolates smoothly between all three geometries of constant curvature.

Original:  H YPERBOLIC & S PHERICAL GCN S
Revised:  κ -GCN S

Original: We start by introducing the methods upon which we build.
Revised: We start by introducing the methods upon which we build.

Original: For space reasons, we will here detail just our proposed hyperbolic GCN.
Revised: We present our models for spaces of constant sectional curvature, in the κ -stereographic model.

Original: However, generalizations tospherical and cartesian products of constant curvature spaces Gu et al. (2019) are straightforward given the previous discussion about universality of gyrovector spaces.
Revised: However, the generalization to cartesian products of such spaces (Gu et al., 2019) follows naturally from these tools.

Original: 3. G RAPH C ONVOLUTIONAL N ETWORKS
Revised: 3. G RAPH C ONVOLUTIONAL N ETWORKS

Original: The problem of node classiﬁcation on a graph has long been tackled with explicit regularization using the graph Laplacian Weston et al.
Revised: The problem of node classiﬁcation on a graph has long been tackled with explicit regularization using the graph Laplacian (Weston et al., 2012).

Original: Namely, for a directed graph with adjacency matrix
Revised: Namely, for a directed graph with adjacency matrix

Original: A , by adding the following term to the loss: (cid:80) i,j A ij (cid:107) f
Revised: A , by adding the following term to the loss: (cid:80) i,j A ij (cid:107) f

Original: ( x i ) − f ( x j ) (cid:107) 2 = f
Revised: ( x i ) − f ( x j ) (cid:107) 2 = f

Original: ( X ) T L f ( X ) , where
Revised: ( X ) T L f ( X ) , where

Original: L = D − A is the (unnormalized) graph Laplacian, D ii :
Revised: L = D − A is the (unnormalized) graph Laplacian, D ii :

Original: =
Revised: =

Original: (cid:80) k
Revised: (cid:80) k

Original: A ik deﬁnes the (diagonal) degree matrix, f contains the trainable parameters of the model and X =
Revised: A ik deﬁnes the (diagonal) degree matrix, f contains the trainable parameters of the model and X =

Original: ( x ji ) ij the node features of the model.
Revised: ( x ji ) ij the node features of the model.

Original: Such a regularization is expected to improve generalization if connected nodes in the graph tend to share labels; node i with feature vector x
Revised: Such a regularization is expected to improve generalization if connected nodes in the graph tend to share labels; node i with feature vector x

Original: i is represented as f ( x i ) in a Euclidean space.
Revised: i is represented as f ( x i ) in a Euclidean space.

Original: With the aim to obtain more scalable models, Kipf et al.  propose to make this regularization implicit by incorporating it into what they call graph convolutional networks Kipf & Welling (2016), which they motivate as a ﬁrst order approximation of spectral graph convolutions, yielding the following  layer architecture (detailed in appendix A):
Revised: With the aim to obtain more scalable models, Defferrard et al. (2016); Kipf & Welling (2017) propose to make this regularization implicit by incorporating it into what they call graph convolutional networks (GCN), which they motivate as a ﬁrst order approximation of spectral graph convolutions, yielding the following scalable layer architecture (detailed in appendix A):

Original: [Equation]
Revised: [Equation]

Original:  where ˜ A =
Revised: ) where ˜ A =

Original: A + I has added self-connections, ˜ D ii
Revised: A + I has added self-connections, ˜ D ii

Original: =
Revised: =

Original: (cid:80) k
Revised: (cid:80) k

Original: ˜
Revised: ˜

Original: A ik deﬁnes its diagonal degree matrix, σ is a non-linearity such as sigmoid, tanh or ReLU = max(0 , · ) , and W ( t ) and H ( t ) are the parameter and activation matrices of layer t respectively, with H (0) =
Revised: A ik deﬁnes its diagonal degree matrix, σ is a non-linearity such as sigmoid, tanh or ReLU = max(0 , · ) , and W ( t ) and H ( t ) are the parameter and activation matrices of layer t respectively, with H (0) =

Original: X the input feature matrix.
Revised: X the input feature matrix.

Original: 3. T OOLS FOR A H YPERBOLIC GCN
Revised: 3. T OOLS FOR A κ -GCN

Original: Learning a parametrized function f θ that respects hyperbolic geometry has been studied in Ganea et al.
Revised: Learning a parametrized function f θ that respects hyperbolic geometry has been studied in (Ganea et al., 2018a): neural layers and hyperbolic softmax.

Original: (2018b).
Revised: We generalize their deﬁnitions into the κ -stereographic model, unifying operations in positive and negative curvature. We explain how curvature introduces a fundamental difference between left and right matrix multiplications, depicting the Möbius matrix multiplication of (Ganea et al., 2018a) as a right multiplication, independent for each embedding.

Original: More precisely, the authors consider feed forward neural networks as well as recurrent neural networks and derive hyperbolic counterparts to essential building blocks such as dense layers and logits in hyperbolic space. We leverage their results in order to generalize Graph Neural Networks.
Revised: We then introduce a left multiplication by extension of gyromidpoints which ties the embeddings, which is essential for graph neural networks.

Original: 3. H YPERBOLIC R IGHT M ATRIX M ULTIPLICATION
Revised: 3. κ -R IGHT -M ATRIX -M ULTIPLICATION

Original: The ﬁrst needed ingredient is a deﬁnition of right matrix multiplication XW , where X ∈ R n × d denotes the embedding and  W ∈ Rd × e are the weights. We assume that each row X i • ∈ P
Revised: Let X ∈ R n × d denote a matrix whose n rows are d -dimensional embeddings in st dκ , and let W ∈ R

Original: d holds the hyperbolic embedding of node i and W is an Euclidean weight matrix.
Revised: d × e denote a weight matrix.

Original: Let us ﬁrst understand what a right matrix multiplication isdoing in Euclidean space. The Euclidean right multiplication can be written column-wise ( XW )• i as follows:
Revised: Let us ﬁrst understand what a right matrix multiplication is in Euclidean space: the Euclidean right multiplication can be written row-wise as ( XW ) i •

Original: [Equation] This leads to a new feature ( XW ) • i by linear combining the features of X using the weights
Revised: =

Original: W 1 i , . . .
Revised: X i • W .

Original: , W di . The features viewed as vectors X • j are not elements of hyperbolic space but rather elements of R d .
Revised: Hence each d -dimensional Euclidean embedding is modiﬁed independently by a right matrix multiplication.

Original: It hence makes sense to lift this operation to the [-tangent space T 
 P d ∼ = R d , as suggested by Ganea et al. (2018b):-]
Revised: A natural adaptation of this operation to the [-tangent space T 
 P d ∼ = R d , as suggested by Ganea et al. (2018b):-]κ -stereographic model yields the following deﬁnition.

Original: Deﬁnition 3.1.
Revised: Deﬁnition 1.

Original: Given an embedding X ∈ R n × d holding hyperbolic embeddings in its rows and weights W ∈ R d × e , the hyperbolic right matrix multiplication is deﬁned  as
Revised: Given a matrix X ∈ R n × d holding κ -stereographic embeddings in its rows and weights W ∈ R d × e , the κ -right-matrix-multiplication is deﬁned row-wise as

Original: [Equation]
Revised: [Equation]

Original: where exp c 0 and log c 0 denote the exponential and logarithmic map in the Poincaré model.
Revised: where exp κ 0 and log κ 0 denote the exponential and logarithmic map in the κ -stereographic model.

Original: This deﬁnition is in perfect agreement with the hyperbolic scalar multiplication, which can also be written as r ⊗ M x = exp c 0 ( r log c 0 ( x )) .This multiplication has the following desirable properties Ganea et al. (2018b).
Revised: This deﬁnition is in perfect agreement with the hyperbolic scalar multiplication for κ < 0 , which can also be written as r ⊗ κ x = exp κ 0 ( r log κ 0 ( x )) .

Original: Another important element of any feed forward neural network is the application of a non-linearity . This  is solved in a similar fashion in Ganea et al. (2018b).
Revised: This operation is known to have desirable properties such as associativity (Ganea et al., 2018a).

Original: Given any map σ : R d −→ R d , x (cid:55)→ σ ( x ) , we deﬁne its hyperbolic dual as σ M : P d −→ P d , x (cid:55)→ exp c 0 ( σ (log c 0 ( x ))) .

Original: H YPERBOLIC L EFT M ATRIX M ULTIPLICATION , L INEAR C OMBINATIONS AND M IDPOINTS
Revised: 3. κ -L EFT -M ATRIX -M ULTIPLICATION AS A M IDPOINT E XTENSION

Original: For graph neural networks we also need the notion of message passing among neighboring nodes. We hence need an operation that combines different node embeddings and is permutation invariant.
Revised: For graph neural networks we also need the notion of message passing among neighboring nodes, i.e. an operation that combines / aggregates the respective embeddings together.

Original: In Euclidean space this operation was achieved by the left multiplication of the embedding with the preprocessed adjacency ˆ A : H ( l +1)
Revised: In Euclidean space such an operation is given by the left multiplication of the embeddings matrix with the (preprocessed) adjacency ˆ A : H ( l +1)

Original: =
Revised: =

Original: σ (cid:16) ˆ AZ ( l ) (cid:17) where Z ( l ) =
Revised: σ ( ˆ AZ ( l ) ) where Z ( l ) =

Original: H ( l ) W ( l )  Let us consider the left multiplication in more detail.
Revised: H ( l ) W ( l ) . Let us consider this left multiplication.

Original: Forease of notation we denote the preprocessed adjacency as A ∈ R n × n . The matrix product is then
Revised: For A ∈ R n × n , the matrix product is given row-wise by:

Original: [Equation]
Revised: [Equation]

Original: This means that the new representation of node i is obtained by calculating the linear combination of all the other node embeddings, weighted by the i -th row of A .Notice that we have already seen this when studying the architecture of GCN on the node level.
Revised: This means that the new representation of node i is obtained by calculating the linear combination of all the other node embeddings, weighted by the i -th row of A .

Original: To adapt GCNs to hyperbolic space we hence need a notion oftaking a weighted linear combinationor barycentric coordinates .
Revised: An adaptation to the κ -stereographic model hence requires a notion of weighted linear combination .

Original: Obviously, the Euclidean operation will not sufﬁce as it is not respecting the underlying geometry nor restricted to P dc .
Revised: We propose such an operation in st dκ by performing a κ -scaling of a gyromidpoint − whose deﬁnition is reminded below. Indeed, in

Original: In order to introduce a hyperbolic linear combination, let us ﬁrst look at the  Euclidean version for only two vectors x, y ∈ R d and weights α, β ∈ R :z = αx + βy
Revised: Euclidean space, the weighted linear combination α x + β y can be re-written as ( α + β ) m E ( x , y ; α, β ) with Euclidean midpoint m E ( x , y ; α, β ) :

Original: This operation is visualized in ﬁg. 3: The resulting sum αx + βy is actually the weighted midpoint m E ( x, y ; α, β ) =
Revised: =

Original: αα + β x + βα + β yscaled by the factor α + β .
Revised: αα + β x + βα + β y .

Original: Scaling a vector in Euclidean space has the effect that the length of the vector is scaled by the same factor but the direction remains unchanged.
Revised: This motivates generalizing the above operation to st dκ as follows. Deﬁnition 2. Given a matrix X ∈ R n × d holding κ -stereographic embeddings in its rows and weights A ∈ R n × n , the κ -left-matrix-multiplication is deﬁned row-wise as [Equation]

Original: We can achieve the same scaling behaviour in hyperbolic space using the hyperbolic scalar multiplication ⊗ c M since it has the similar property
Revised: The κ -scaling is motivated by the fact that d κ ( 0 , r ⊗ κ x ) = |

Original: || r ⊗ c M x || M = d M c (0 ,r ⊗ c M x ) = r || x || M .
Revised: r | d κ ( 0 , x ) for all r ∈ R , x ∈ st dκ .

Original: Since geodesics through 0 are straight lines in the Poincaré model we also are not changing the direction. We hence reduced the problem to ﬁnding a notion of a weighted midpoint inhyperbolic space. We will use the notion of a weighted midpoint arising from gyro theory, introduced in Ungar (2010):
Revised: We remind that the gyromidpoint is deﬁned when κ ≤ 0 in the κ -stereographic model as (Ungar, 2010): [Equation]

Original: Deﬁnition 3.2. For { x 1 , . . . , x n } ⊂ P dc and weights { α 1 , . . .,
Revised: with λ κ x = 2 / (1 + κ (cid:107) x (cid:107) 2 ) .

Original: α n } ⊂ R the weighted gyromidpoint in the Poincaré model is given by
Revised: Whenever κ > 0 , we have to further require the following condition:

Original: [Equation]
Revised: [Equation]

Original: Note that this deﬁnition of weighted midpoint implicitly performs a normalization by the term (cid:80) ni
Revised: For two points, one can calculate that ( λ κ x − 1) + ( λ κ y − 1) = 0 is equivalent to κ (cid:107) x (cid:107)(cid:107)

Original: [-=
 α i and we will also assume this for the Euclidean weighted mean.-]
Revised: [-=
 α i and we will also assume this for the Euclidean weighted mean.-]y (cid:107) = 1 , which holds in particular whenever x = − y / ( κ (cid:107) y (cid:107) 2 ) .

Original: As can be seen in ﬁg. 3, the gyromidpoints respect hyperbolic geometry as they follow the geodesics.
Revised: See ﬁg. 3 for illustrations of gyromidpoints.

Original: This midpoint also has some analogous properties to the Euclidean version, as seen in Ungar (2010):
Revised: Our operation (cid:2) κ satisﬁes interesting properties, proved in Appendix C.2.4:

Original: Lemma 1.
Revised: Theorem 4 (Neuter element & κ -scalar-associativity) .

Original: For { x 1 , . . . , x n } ⊂ P dc and y ∈P dc and weights { α 1 . . α n } ⊂ R it holds:
Revised: We have I n (cid:2) κ X = X , and for r ∈ R ,

Original: • m c M ( y ⊕ c M x 1 , . .
Revised: [Equation]

Original: , y ⊕ c M x n ; α 1 , . . . α n ) = y ⊕ c M m c M ( x 1 , . . . , x n ; α 1 , . . .
Revised: The matrix A .

Original: α n ) • d c M ( x 1 , m c M ( x 1 , x 2 ; 12 , 12 )) = 12 d c M ( x 1 , x 2 ) • m c M ( x 1 , . . , x n ; α 1 , . . . α n ) c −→ 0 −−−→ m E ( x 1 , . .
Revised: In most graph neural networks, the matrix A is intented to be a preprocessed adjacency matrix, i.e. renormalized by the diagonal degreee matrix D ii

Original: , x n ; α 1 , . . . α n )
Revised: = (cid:80) k

Original: We are now ready to deﬁne the hyperbolic linear combination in the Poincaré model. Deﬁnition 3.3. For { x 1 , . . . , x n } ⊂ P dc and weights { α 1 , . . .,
Revised: A ik .

Original: α n } ⊂ R , the hyperbolic linear combination in the Poincaré model is given by: ( (cid:80) ni =1 α i ) ⊗ c M m c M ( x 1  , . . . , x n ; α 1, . . . α n ) .
Revised: This normalization is often taken either (i) to the left: D − 1 A , (ii) symmetric: D − 12 AD − 12 or (iii) to the right: AD − 1 .

Original: We can further introduce the left matrix multiplication in the Poincaré model: Deﬁnition 3.4. Given X ∈ R n × d with rows X i • ∈ P dc and A ∈ R n × n , the hyperbolic left matrix multiplication Z
Revised: Note that the latter case makes the matrix right-stochastic 1 , which is a property that is preserved by matrix product and exponentiation.

Original: =
Revised: For this case, we prove the following result in Appendix C.2.5:

Original: A ⊗ c M X is deﬁned row-wise via Z i • = ( n (cid:88) j =
Revised: Theorem 5 ( κ -left-multiplication by right-stochastic matrices is intrinsic) .

Original:  A 
Revised: If A , B are rightstochastic, φ is a isometry of st dκ and X , Y are two matrices holding κ -stereographic embeddings: [Equation] The above result means that A can easily be preprocessed as to make its κ -left-multiplication intrinsic to the metric space ( st dκ , d κ ). At this point, one could wonder: does there exist other ways to take weighted centroids on a Riemannian manifold?

Original: [Equation]
Revised: We comment on two plausible alternatives. Fréchet/Karcher means. They are obtained as arg min x (cid:80) i

Original: ) ⊗
Revised: α i

Original: [Equation]
Revised: d κ ( x , x i ) 2 ; note that although they are also intrinsic, they usually require solving an optimization problem which can be prohibitively expensive, especially when one requires gradients to ﬂow through the solution − moreover, for the space st dκ , it is known that the minimizer is unique if and only if κ ≥ 0 . Tangential aggregations. They are deﬁned by lifting the points in a chosen tangent space via the logarithmic map, performing a linear combination and then projecting back via the exponential map, and were in particular used in the recent works of Chami et al. (2019) and Liu et al.

Original: Lemma 2.
Revised: The below theorem describes that for the κ -stereographic model, this operation is also intrinsic, i.e. commutes with isometries. We prove it in Appendix C.2.6. Theorem 6 (Tangential aggregation is intrinsic) . Deﬁne the tangential aggregation of x 1 , . . .

Original: Given X ∈ R n× d with rows X i • ∈ P
Revised: , x n ∈ st dκ w.r.t.

Original: dc , A ∈ R n × nand r ∈ R , the hyperbolic left matrix multiplication satisﬁes:
Revised: weights { α i } 1 ≤ i ≤ n , at point x ∈ st dκ (for x i (cid:54) = − x / ( κ (cid:107) x (cid:107) 2 ) if κ > 0 ) by: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: For any isometry φ of st dκ , we have

Original: [Equation]
Revised: [Equation]

Original: H YPERBOLIC L OGITS
Revised: 3. L OGITS

Original: Finally, we need the logit and softmax layer, a neccessity for any classiﬁcation task.
Revised: Finally, we need the logit and softmax layer, a neccessity for any classiﬁcation task.

Original: We here use the model of Ganea et al. (2018b).
Revised: We here use the model of (Ganea et al., 2018a), which was obtained in a principled manner for the case of negative curvature.

Original: More details are given in  appendix E.
Revised: We leave for future work the adaptation of their analysis to positive curvature and use in our experiments the straightforwardly adaptated formula to positive curvature, which we detail appendix D.

Original: 3. H YPERBOLIC GCN
Revised: 3. κ -GCN

Original: We are now ready to introduce a hyperbolic version of GCN Kipf & Welling (2017), denoted by H c -GCN.
Revised: We are now ready to introduce our κ -stereographic GCN (Kipf & Welling, 2017), denoted by κ -GCN 2 .

Original: Assume we are given a graph with node level features G
Revised: Assume we are given a graph with node level features G

Original: =
Revised: =

Original: ( V, A , X ) where X ∈ R n × d with each row 
Revised: ( V, A , X ) where X ∈ R n × d with each row X

Original: X i • ∈ P dc and adjacency A ∈ R n × n .
Revised: i • ∈ st dκ and adjacency A ∈ R n × n .

Original: We ﬁrst perform a preprocessing step by mapping theEuclidean features to the Poincaré ball via the projection X (cid:55)→
Revised: We ﬁrst perform a preprocessing step by mapping the

Original:  X 2 √ c || X || max  , where || X || max denotes the maximal  norm among all hyperbolic embeddings in X .
Revised: Euclidean features to st dκ via the projection X (cid:55)→ X / (2 (cid:112) | κ ||| X || max ) , where || X || max denotes the maximal Euclidean norm among all stereographic embeddings in X .

Original: For l ∈ { 0 , . . .
Revised: For l ∈ { 0 , . . .

Original: , L − 2 } , the ( l + 1) -th layer of H c − GCN is given by
Revised: , L − 2 } , the ( l + 1) -th layer of κ -GCN is given by:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where H (0) =
Revised: where H (0) =

Original: X , σ  is some non-linearity  and ˆ A =
Revised: X , σ ⊗ κ ( x ) := exp κ 0 ( σ (log κ 0 ( x ))) is the Möbius version (Ganea et al., 2018a) of a pointwise non-linearity σ and ˆ A =

Original: ˜ D − 12 ˜
Revised: ˜ D − 12 ˜

Original: A ˜ D − 12 .
Revised: A ˜ D − 12 .

Original: The ﬁnal layer is a hyperbolic logit layer:
Revised: The ﬁnal layer is a κ -logit layer (appendix D):

Original: [Equation]
Revised: [Equation]

Original:  A very important property of H c − GCN is that its architecture recovers the Euclidean GCN when we let curvature go to zero:H c -GCN c −→ 0 −−−→ GCN. 3. GCN S IN P RODUCT OF C ONSTANT C URVATURE S PACES The extension of the above approach to spherical spaces and products of spaces of constant curvature is rather straightforward.
Revised: , (16) where W ( L − 1) contains the parameters a k and p k of the κ -logits layer. A very important property of κ -GCN is that its architecture recovers the Euclidean GCN when we let curvature go to zero:

Original: For clarity, we detail it in appendix F.
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the architectures introduced in the previous sections on the tasks of node classiﬁcation and minimizing embedding distortion for several synthetic as well as real datasets.
Revised: We evaluate the architectures introduced in the previous sections on the tasks of node classiﬁcation and minimizing embedding distortion for several synthetic as well as real datasets.

Original: More details are give in appendix G.
Revised: We detail the training setup and model architecture choices to appendix E. [Table]

Original: Minimizing Distortion Our ﬁrst goal is to evaluate the graph embeddings learned by our GCN models on the representation task of ﬁtting the graph metric in the embedding space.
Revised: Distortion Our ﬁrst goal is to evaluate the graph embeddings learned by our GCN models on the representation task of ﬁtting the graph metric in the embedding space.

Original: We desire to minimize the average distortion deﬁned similarly as in Gu et al.
Revised: We desire to minimize the average distortion, i.e. deﬁned similarly as in (Gu et al., 2019): 1 n 2 (cid:80) i,j (cid:18)(cid:16) d ( [Equation] [Equation]

Revised: (cid:17)

Original: [Equation]
Revised: [Equation]

Original: n
Revised: − 1 (cid:19) , where d ( x i , x j ) is the distance between the embeddings of nodes i and j

Original: [Equation]
Revised: [Table]

Original: where d ( x i , x j ) is the distance between the embeddings of nodes i and j, while d G ( i, j ) is their graph distance (shortest path length).
Revised: while d G ( i, j ) is their graph distance (shortest path length).

Original: We create three synthetic datasets as follows. Tree: a balanced tree of depth 5 and branching factorconsisting of 1365 nodes and 2728 edges. 
Revised: We create three synthetic datasets that best reﬂect the different geometries of interest: i) “Tree‘”: a balanced tree of depth 5 and branching factor 4 consisting of 1365 nodes and 1364 edges. ii) “Torus”:

Original: Torus: We sample points (nodes) from the (planar) torus, i.e. from the unit connected square; two nodes are connected by an edge iff their toroidal distance (the warped distance) is smaller than a ﬁxed R = 0.01; this gives 1000 nodes and 30626 edges.
Revised: We sample points (nodes) from the (planar) torus, i.e. from the unit connected square; two nodes are connected by an edge iff their toroidal distance (the warped distance) is smaller than a ﬁxed R = 0.01; this gives 1000 nodes and 30626 edges.

Original: Spherical Graph: we sample points (nodes) from S 2 , connecting nodes iff their distance is smaller than 0.2, leading to 1000 nodes and 17640 edges.
Revised: iii) “Spherical Graph”: we sample points (nodes) from S 2 , connecting nodes iff their distance is smaller than 0.2, leading to 1000 nodes and 17640 edges.

Original: For the GCN models, we use 1-hot initial node features.
Revised: For the GCN models, we use 1-hot initial node features.

Original: We use two GCN layers with dimensionsand 10.
Revised: We use two GCN layers with dimensionsand 10.

Original: The non-Euclidean models do not use additional non-linearities between layers.
Revised: The non-Euclidean models do not use additional non-linearities between layers.

Original: All euclidean parameters are updated using the ADAM optimizer with learning rate 0.01.
Revised: All euclidean parameters are updated using the ADAM optimizer with learning rate 0.01.

Original: Curvatures are learned using (stochastic) gradient descent and learning rate of 0.0001.
Revised: Curvatures are learned using (stochastic) gradient descent and learning rate of 0.0001.

Original: All models are trained forepochs
Revised: All models are trained forepochs

Original: and we report the minimal achieved distortion.
Revised: and we report the minimal achieved distortion.

Original: The results shown in table 1 reveal the beneﬁt of our models.
Revised: The results shown in table 1 reveal the beneﬁt of our models.

Original: One can notice that estimated curvatures correspond to our geometric knowledge about these speciﬁc datasets.[Table]
Revised: One can notice that estimated curvatures correspond to our geometric knowledge about these speciﬁc datasets.

Original:  N ODE C LASSIFICATION
Revised: 4. N ODE C LASSIFICATION

Original: We consider the popular node classiﬁcation datasets Citeseer Sen et al. , Cora-ML McCallum et al. and Pubmed Namata et al. (2012) which are citation networks.
Revised: We consider the popular node classiﬁcation datasets Citeseer (Sen et al., 2008), Cora-ML (McCallum et al., 2000) and Pubmed (Namata et al., 2012).

Original: Node labels correspond to the particular subﬁeld the published document is associated with.
Revised: Node labels correspond to the particular subﬁeld the published document is associated with.

Original: Dataset statistics and training details are deffered to the appendix G due to the lack of space.
Revised: Dataset statistics and splitting details are deferred to the appendix E due to the lack of space.

Original: Curvature Estimations of Datasets To understand how far are the real graphs of the above datasets from the Euclidean geometry, we ﬁrst estimate the graph curvature of the four studied datasets using the deviation from the Parallelogram Law Gu et al. This is detailed in appendix H. Curvature histograms are shown in ﬁg. 4.
Revised: Curvature Estimations of Datasets To understand how far are the real graphs of the above datasets from the Euclidean geometry, we ﬁrst estimate the graph curvature of the four studied datasets using the deviation from the Parallelogram Law (Gu et al., 2019) as detailed in appendix F. Curvature histograms are shown in ﬁg. 4.

Original: It can be noticed that the datasets are mostly non-Euclidean.
Revised: It can be noticed that the datasets are mostly non-Euclidean, thus offering a good motivation to apply our constant-curvature GCN architectures. [Table]

Original: Thus, we have a good motivation to apply our constant-curvature GCN architectures.
Revised: Training Details We trained the Euclidean models with the hyperparameters chosen as reported in (Klicpera et al., 2019). Namely, for GCN we use one hidden layer of size 64 , dropout on the embeddings and the adjacency of rate 0 . 5 as well as L 2 -regularization for the weights of the ﬁrst layer with λ = 0 . Only for Cora-ML we had to adjust the regularization factor λ to 0 .

Original: Node classiﬁcation results.
Revised: 002 to ensure similar scores as achieved in (Klicpera et al., 2019). All Non-Euclidean models use biased-L2 regularization with α = 10 and λ = 2 e − 2 . Euclidean models used L2 regularization with the same parameter λ . We used a combination of dropout and dropconnect for the non-Euclidean models.

Revised: All models have the same number of parameters.

Revised: We use 2 GCN layers, hidden dimension 64.

Revised: Product models split hidden dimension into [32, 32] and also input features equally.

Revised: Non-Euclidean models do not use additional non-linearities.

Revised: Euclidean parameters use a learning rate of 0.01 for all models using ADAM.

Revised: The curvatures are learned using gradient descent with a learning rate of 0.01.

Revised: We show the values of the learned curvatures in appendix E. We use early stopping: we ﬁrst train for a maximum of 2000 epochs, then we check every 200 epochs for improvement in the validation cross entropy loss; if that is not observed, we stop.

Revised: Node classiﬁcation results.

Original: These are shown in table 2.
Revised: These are shown in table 2.

Original: It can be seen that our models sometimes outpeform the two Euclidean GCN considered (with or without non-linearities), showcasing the beneﬁt of our proposed architecture.
Revised: It can be seen that our models are competitive with the two Euclidean GCN considered (with or without non-linearities), showcasing the beneﬁt of our proposed architecture.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduced  natural extensions of graph neural networks to the non-Euclidean setting. To this end [Table] we studied product of spaces of constant sectional curvature and equipped them with
Revised: In this paper, we introduced a natural extension of graph convolutional networks to the stereographic models of both positive and negative curvatures in a uniﬁed manner.

Original: Model [Table]
Revised: We show how this choice of models permits to smoothly interpolate between positive and negative curvature, allowing the curvature of the model to be trained independent of an initial sign choice.

Original: additional structure in the form of gyrovector spaces and matrix multiplications. Empirically we showed the beneﬁt of our model on several real and synthetic datasets.
Revised: We hope that our models will open new exciting directions into non-Euclidean graph neural networks.

Original: A. C ONVOLUTIONAL N EURAL N ETWORKS ON G RAPHS
Revised: A. C ONVOLUTIONAL N EURAL N ETWORKS ON G RAPHS

Original: One of the pioneering works on neural networks in non-Euclidean domains was done by Defferrard et al.
Revised: One of the pioneering works on neural networks in non-Euclidean domains was done by (Defferrard et al., 2016).

Original: Their idea was to extend convolutional neural networks for graphs using tools from graph signal processing .
Revised: Their idea was to extend convolutional neural networks for graphs using tools from graph signal processing .

Original: Given a graph G =
Revised: Given a graph G =

Original: ( V, A ) , where A is the adjacency matrix and V is a set of nodes, we deﬁne a signal on the nodes of a graph to be a vector x ∈ R n where x i is the value of the signal at node i .
Revised: ( V, A ) , where A is the adjacency matrix and V is a set of nodes, we deﬁne a signal on the nodes of a graph to be a vector x ∈ R n where x i is the value of the signal at node i .

Original: Consider the diagonalization of the symmetrized graph Laplacian ˜ L = U Λ U T , where Λ = diag ( λ 1 , . .
Revised: Consider the diagonalization of the symmetrized graph Laplacian ˜ L = U Λ U T , where Λ = diag ( λ 1 , . .

Original: . , λ n ) .
Revised: . , λ n ) .

Original: The eigenbasis U allows to deﬁne the graph Fourier transform ˆ x = U T x ∈ R n .
Revised: The eigenbasis U allows to deﬁne the graph Fourier transform ˆ x = U T x ∈ R n .

Original: In order to deﬁne a convolution for graphs, we shift from the vertex domain to the Fourier domain :
Revised: In order to deﬁne a convolution for graphs, we shift from the vertex domain to the Fourier domain :

Original: [Equation]
Revised: [Equation]

Original: Note that ˆ x = U T x and ˆ y =
Revised: Note that ˆ x = U T x and ˆ y =

Original: U
Revised: U

Original: T
Revised: T

Original: y are the graph Fourier representations and we use the element-wise product (cid:12) since convolutions become products in the Fourier domain.
Revised: y are the graph Fourier representations and we use the element-wise product (cid:12) since convolutions become products in the Fourier domain.

Original: The left multiplication with U maps the Fourier representation back to a vertex representation.
Revised: The left multiplication with U maps the Fourier representation back to a vertex representation.

Original: As a consequence, a signal x ﬁltered by g θ becomes y
Revised: As a consequence, a signal x ﬁltered by g θ becomes y

Original: =
Revised: =

Original: U g θ (Λ)
Revised: U g θ (Λ)

Original: U T x where g θ = diag ( θ ) with θ ∈ R n constitutes a ﬁlter with all parameters free to vary.
Revised: U T x where g θ = diag ( θ ) with θ ∈ R n constitutes a ﬁlter with all parameters free to vary.

Original: In order to avoid the resulting complexity
Revised: In order to avoid the resulting complexity

Original: O ( n ) , Defferrard et al. (2016) replace the non-parametric ﬁlter by a polynomial ﬁlter:
Revised: O ( n ) , (Defferrard et al., 2016) replace the non-parametric ﬁlter by a polynomial ﬁlter:

Original: [Equation]
Revised: [Equation]

Original: where θ ∈ R K resulting in a complexity
Revised: where θ ∈ R K resulting in a complexity

Original: O ( K ) .
Revised: O ( K ) .

Original: Filtering a signal is unfortunately still expensive since y =
Revised: Filtering a signal is unfortunately still expensive since y =

Original: U g θ (Λ) U T x requires the multiplication with the Fourier basis U , thus resulting in complexity O ( n 2 ) .
Revised: U g θ (Λ) U T x requires the multiplication with the Fourier basis U , thus resulting in complexity O ( n 2 ) .

Original: As a consequence, Defferrard et al.
Revised: As a consequence, (Defferrard et al., 2016) circumvent this problem by choosing the

Original: (2016) circumvent this problem by choosing the Chebyshev polynomials T k as a polynomial basis, g θ (Λ) =
Revised: Chebyshev polynomials T k as a polynomial basis, g θ (Λ) =

Original: (cid:80) Kk =

Revised: (cid:80) Kk =

Original: θ k T k (˜Λ) where ˜Λ = 2Λ λ max − I .
Revised: θ k T k (˜Λ) where ˜Λ = 2Λ λ max − I .

Original: As a consequence, the ﬁlter operation becomes y = (cid:80) Kk
Revised: As a consequence, the ﬁlter operation becomes y = (cid:80) Kk

Original: =

Revised: =

Original: θ k T k
Revised: θ k T k

Original: (ˆ L ) x
Revised: (ˆ L ) x

Original: where ˆ L = 2 L λ max − I .
Revised: where ˆ L = 2 L λ max − I .

Original: This led to a K - localized ﬁlter since it depended on the K -th power of the Laplacian.
Revised: This led to a K - localized ﬁlter since it depended on the K -th power of the Laplacian.

Original: The recursive nature of these polynomials allows for an efﬁcient ﬁltering of complexity
Revised: The recursive nature of these polynomials allows for an efﬁcient ﬁltering of complexity

Original: O
Revised: O

Original: ( K | E | ) , thus leading to an computationally appealing deﬁnition of convolution for graphs.
Revised: ( K | E | ) , thus leading to an computationally appealing deﬁnition of convolution for graphs.

Original: The model can also be built in an analogous way to CNNs, by stacking multiple convolutional layers, each layer followed by a non-linearity.
Revised: The model can also be built in an analogous way to CNNs, by stacking multiple convolutional layers, each layer followed by a non-linearity.

Original: A. G RAPH C ONVOLUTIONAL N ETWORKS
Revised: A. G RAPH C ONVOLUTIONAL N ETWORKS

Original: Kipf & Welling (2017) extended the work of Defferrard et al. and inspired many follow-up architectures Chen et al. Hamilton et al. ; Abu-El-Haija et al.
Revised: (Kipf & Welling, 2017) extended the work of (Defferrard et al., 2016) and inspired many follow-up architectures (Chen et al., 2018; Hamilton et al., 2017; Abu-El-Haija et al., 2018; Wu et al., 2019).

Original: ; Wu et al. The core idea of Kipf & Welling (2017) is to limit each ﬁlter to 1-hop neighbours by setting K = 1 , leading to a convolution that is linear in the Laplacian ˆ L :
Revised: The core idea of (Kipf & Welling, 2017) is to limit each ﬁlter to 1-hop neighbours by setting K = 1 , leading to a convolution that is linear in the Laplacian ˆ L :

Original: [Equation]
Revised: [Equation]

Original: They further assume λ max ≈ 2 , resulting in the expression
Revised: They further assume λ max ≈ 2 , resulting in the expression

Original: [Equation]
Revised: [Equation]

Original: To additionally alleviate overﬁtting, Kipf & Welling (2017) constrain the parameters as θ 

Revised: To additionally alleviate overﬁtting, (Kipf & Welling, 2017) constrain the parameters as θ

Original: = − θ 1 = θ , leading to the convolution formula
Revised: = − θ 1 = θ , leading to the convolution formula

Original: [Equation]
Revised: [Equation]

Original: Since I + D − 12 AD − 12 has its eigenvalues in the range [0 , 2] , they further employ a reparametrization trick to stop their model from suffering from numerical instabilities:
Revised: Since I + D − 12 AD − 12 has its eigenvalues in the range [0 , 2] , they further employ a reparametrization trick to stop their model from suffering from numerical instabilities:

Original: [Equation]
Revised: [Equation]

Original: where ˜ A =
Revised: where ˜ A =

Original: A + I and ˜ D ii
Revised: A + I and ˜ D ii

Original: = (cid:80) nj =1 ˜
Revised: = (cid:80) nj =1 ˜

Original: A ij .
Revised: A ij .

Original: Rewriting the architecture for multiple features X ∈ R n × d 1 and parameters Θ ∈ R d 1 × d 2 instead of x ∈ R n and θ ∈ R , gives
Revised: Rewriting the architecture for multiple features X ∈ R n × d 1 and parameters Θ ∈ R d 1 × d 2 instead of x ∈ R n and θ ∈ R , gives

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal model consists of multiple stacks of convolutions, interleaved by a non-linearity σ :
Revised: The ﬁnal model consists of multiple stacks of convolutions, interleaved by a non-linearity σ :

Original: [Equation]
Revised: [Equation]

Original: where H (0) =
Revised: where H (0) =

Original: X and Θ ∈ R n × d k .
Revised: X and Θ ∈ R n × d k .

Original: The ﬁnal output H ( K ) ∈ R n × d K represents the embedding of each node i as h
Revised: The ﬁnal output H ( K ) ∈ R n × d K represents the embedding of each node i as h

Original: i =
Revised: i =

Original: H i • ∈ R d K and can be used to perform node classiﬁcation:
Revised: H i • ∈ R d K and can be used to perform node classiﬁcation:

Original: [Equation]
Revised: [Equation]

Original: where W ∈ R d K × L , with L denoting the number of classes.
Revised: where W ∈ R d K × L , with L denoting the number of classes.

Original: In order to illustrate how embeddings of neighbouring nodes interact, it is easier to view the architecture on the node level .
Revised: In order to illustrate how embeddings of neighbouring nodes interact, it is easier to view the architecture on the node level .

Original: Denote by N ( i ) the neighbours of node i .
Revised: Denote by N ( i ) the neighbours of node i .

Original: One can write the embedding of node i at layer k + 1 as follows:
Revised: One can write the embedding of node i at layer k + 1 as follows:

Original: [Equation]
Revised: [Equation]

Original: Notice that there is no dependence of the weight matrices Θ ( l ) on the node i , in fact the same parameters are shared across all nodes.
Revised: Notice that there is no dependence of the weight matrices Θ ( l ) on the node i , in fact the same parameters are shared across all nodes.

Original: In order to obtain the new embedding h ( k +1) i of node i , we average over all embeddings of the neighbouring nodes.
Revised: In order to obtain the new embedding h ( k +1) i of node i , we average over all embeddings of the neighbouring nodes.

Original: This Message Passing mechanism gives rise to a very broad class of graph neural networks Kipf & Welling (2017); Veliˇckovi´c et al.
Revised: This Message Passing mechanism gives rise to a very broad class of graph neural networks (Kipf & Welling, 2017;

Original:  Hamilton et al. ; Gilmer et al. (2017); Chen et al. ; Klicpera et al.
Revised: Veli ˇ ckovi ´ c et al., 2018; Hamilton et al., 2017; Gilmer et al., 2017; Chen et al., 2018; Klicpera et al., 2019; Abu-El-Haija et al., 2018).

Original: [Equation] ; Abu-El-Haija et al [Equation] To be more precise, GCN falls into the more general category of models of the form
Revised: To be more precise, GCN falls into the more general category of models of the form

Original: [Equation]
Revised: [Equation]

Original: Models of the above form are deemed Message Passing Graph Neural Networks and many choices for AGGREGATE and COMBINE have been suggested in the literature Kipf & Welling (2017); Hamilton et al. ; Chen et al. B M ANIFOLDS AND R IEMANNIAN G EOMETRY
Revised: Models of the above form are deemed Message Passing Graph Neural Networks and many choices for AGGREGATE and COMBINE have been suggested in the literature (Kipf & Welling, 2017;

Original: In this section, we give an overview over the mathematical tools from differential geometry leveraged in this paper.

Original: More precisely, the notion of a manifold and Riemannian geometry will be discussed with a special emphasis on spherical geometry, hyperbolic space and its various models.

Original: For a deeper treatment of differential geometry we refer to Robbin & Salamon (2011); Spivak (1979); Hopper &

Original: Andrews (2010).

Original: B. M ANIFOLDS AND THEIR S UBCLASSES

Original: Informally stated, manifolds are spaces that locally look like Euclidean space.

Original: An intuitive example is given by the sphere S 2 = { x ∈ R 3 :

Original: || x

Original: || = 1 } .

Original: For someone standing on the Earth, space seems ﬂat and therefore Euclidean.

Original: This only holds locally; globally the Earth obeys a different geometry as one can notice by watching ships disappear towards the horizon.

Original: This resemblance with Euclidean space allows for many translations of important concepts from calculus to more general notions in manifolds.

Original: This locality property can be formalized:

Original: Deﬁnition B.1.

Original: A topological manifold M of dimension n is a Hausdorff, connected, topological space which is locally homeomorphic to R n .

Original: In other words, for every point p ∈ M , one can ﬁnd a neighbourhood V p ⊂ M containing p and a homeomorphism φ , such that U =

Original: φ ( V p ) ⊂ R n .

Original: There exist further subclasses of this deﬁnition of a manifold, each allowing for more additional structure.

Original: The class of manifolds that are mainly of interest in this work are smooth manifolds .

Original: In order to deﬁne this mathematical object, the notion of charts and atlases needs to be introduced.

Original: Deﬁnition B.2.

Original: A chart is a pair ( V, ψ ) where V is an open neighbourhood in M and ψ : V −→

Original: U =

Original: ψ ( V ) is a homeomorphism onto U ⊂ R n .

Original: Deﬁnition B.3.

Original: A collection of charts

Original: A

Original: = { ( U i , ψ i ) } i ∈ I is called C r -atlas for M if: • M = ∪ i ∈ I U
Revised: Hamilton et al., 2017; Chen et al., 2018).

Original: i • The transition functions ψ ij = ψ

Original: i ◦ ψ − 1 j restricted to the intersection U i ∩ U j are C r -differentiable.

Original: With these deﬁnitions at hand, the smooth manifold can be introduced:

Original: Deﬁnition B.4.

Original: A smooth manifold M is a topological manifold equipped with an equivalence class of atlases, whose transition functions are all smooth.

Original: These deﬁnitions are easier digested when accompanied with an example :

Original: Example 1.

Original: The 2-sphere S 2 = { ( x, y, z ) ∈ R 3 : x 2 + y 2 + z 2 = 1 } is an example of a smooth manifold.

Original: One can see this by the following construction of an atlas:

Original: Deﬁne the open set

Original: [Equation]

Original: and the open unit disk in R 2 :

Original: [Equation]

Original: Further deﬁne the projection

Original: [Equation]

Original: with the inverse given by

Original: [Equation]

Original: Both functions are smooth, hence p z constitutes a coordinate chart on S 2 .

Original: We can extend the same arguments to the other open sets U − z , U + x , U − x , U + y , U − y with the appropriate disk and projection chosen to obtain 5 more charts.

Original: The union of these sets cover S 2 and we thus have an atlas.

Original: We hence have proven that S 2 is a topological manifold.

Original: Consider now the transition functions given by f ± i,j = p ± i ◦ ( p ± j ) −

Original: For example

Original: [Equation]

Original: Again, one easily sees that this is a smooth map and one can further verify that this holds for any f ± i,j .

Original: As a result, S 2 is a smooth manifold.

Original: B. R IEMANNIAN M ANIFOLDS

Original: A very powerful subclass of smooth manifolds is given by Riemannian manifolds .

Original: The additional required structure is given by the so-called Riemannian metric .

Original: To understand this concept, we need to ﬁrst visit the notion of tangent space.

Original: B.2. T ANGENT S PACE

Original: In differential geometry, there are usually two ways of looking at manifolds.

Original: The ﬁrst way is viewing the manifold as an object embedded in an ambient space – extrinsic view.

Original: Again, the easiest example is the sphere S 2 embedded in Euclidean space R 3 .

Original: The second view (intrinsic) is studying the manifold as an object itself, without relying on any ambient space.

Original: The second view is the preferred one in abstract differential geometry as it is mathematically more appealing to separate the manifold from its surrounding space and investigate it intrinsically.

Original: For machine learning, the ﬁrst view is to be favored as one wants to use tools from Euclidean geometry which are present in the ambient space.

Original: As a result, there are two ways of introducing tangent spaces and for the aforementioned reasons.

Original: However, only the deﬁnition arising from the extrinsic point of view will be discussed here.

Original: Deﬁnition B.5.

Original: Consider M ⊂ R n a smooth manifold and a point p ∈ M .

Original: A vector v ∈ R n is called tangential to p if and only if there exists a smooth curve γ : R (cid:55)→ M fulﬁlling:

Original: • γ (0) =

Original: p • ˙ γ (0) =

Original: v

Original: We denote by T p M = { ˙ γ (0) : γ : R (cid:55)→ M is smooth, γ (0) = p } the tangent space at p .

Original: This deﬁnition is extrinsic in the sense that it relies on the embedding space R n by letting the tangent vectors "stick out" from the manifold (consider ﬁg. 6 for instance).

Original: We will use the concept of tangent spaces to deﬁne the Riemannian metric.

Original: B.2. R IEMANNIAN M ETRIC

Original: A Riemannian metric induces for every point p ∈ M an inner product in the tangent space T p M , meaning a map g p : T p M × T p M −→ R which is symmetric and positive deﬁnite:

Original: • g p ( x, y ) =

Original: g p

Original: [-( y, x ) for every x, y ∈ T p M • g p ( x, x ) ≥ 
-]
Revised: [-( y, x ) for every x, y ∈ T p M • g p ( x, x ) ≥ 
-]

Original: for every x ∈ T p M

Original: Moreover, the collection of inner products varies smoothly in the location of the tangent space p .

Original: The best known example of a Riemannian metric is given by Euclidean space R n .

Original: The tangent space at p is given by T p R n ∼ = R n and is equipped with the standard inner product g p ( x, y ) =

Original: x T y .

Original: We will usually identify g p with its matrix representation.

Original: The Riemannian metric allows for the extension of many mathematical tools to Riemannian manifolds that are needed in order to study machine learning in non-Euclidean spaces.

Original: B. M ATHEMATICAL TOOLS

Original: Given a Riemannian metric g on a manifold M , one is able to deﬁne many quantities analogous to

Original: Euclidean space.

Original: B.3. D ISTANCE FUNCTION

Original: Given a curve γ : [0 , 1] −→ M , we can use the metric g to calculate the length of it:

Original: [Equation]

Original: Given now two points x, y ∈ M , we can deﬁne the distance between the two by calculating the inﬁmum of the length over all curves γ fulﬁlling γ (0) = x and γ (1) =

Original: y :

Original: [Equation]

Original: Notice that in the example of M = R n , this formula reduces to

Original: [Equation]

Original: where the second equality can be shown by the formalism of Euler-Lagrange.

Original: The curve that achieves the inﬁmum in (2.1) is called geodesic .

Original: It is the natural extension of straight lines in Euclidean space, as they are also the distance minimizing curves between two points.

Original: B.3. G RADIENTS IN THE T ANGENT S PACES OF R IEMANNIAN M ANIFOLDS

Original: In order to be able to perform optimization, we need to compute gradients of a cost function.

Original: Again, we have to move to tangent spaces to overcome this problem.

Original: Given parameters z ∈ M , we want to calculate a gradient that indicates the best direction we should take in order to reduce the value of the cost function L and update the parameters z of our model correspondingly.

Original: One could naively calculate the Euclidean gradient and perform the update

Original: [Equation]

Original: There are two ﬂaws in this scheme:

Original: • The Euclidean gradient does not take into account the geometry of the manifold.

Original: We need the Riemannian gradient , that applies a correction according to the metric tensor:

Original: [Equation]

Original: • The notion of a step is not well-deﬁned on M since there is no guarantee that the result of the subtraction lies again in M .

Original: We hence need a manifold-speciﬁc way of taking a step.

Original: The correct way of taking a step on a manifold is given

Original: by the

Original: exponential

Original: map introduced

Original: in the next section

Original: B.3. E XPONENTIAL AND L OGARITHMIC M AP

Original: Consider a point p ∈ M and v ∈ T p M .

Original: The unit speed geodesic starting from x in direction v is the unique geodesic γ x,v with γ (0) =

Original: x and ˙ γ (0) = v .

Original: The exponential map exp p ( v ) allows to map from the tangent space T p M to the manifold M and is deﬁned as follows:

Original: [Equation]

Original: Its inverse, log p : M (cid:55)→ T p M is called logarithmic map .

Original: Notice that the deﬁnition of the exponential map depends on the unit speed geodesic.

Original: Unfortunately, closed-formulas for the latter are rather seldom or computationally very expensive.

Original: As a result, one sometimes has to rely on cheaper approximations to the exponential map, called retractions .

Original: Finally, observe that for M = R n , it holds that exp x

Original: ( v ) =

Original: x + v and log x ( y ) =

Original: y − x .

Original: The combination of the two previous sections leads to Riemannian optimization Bonnabel (2013).

Original: R IEMANNIAN O PTIMIZATION

Original: Denote by L a loss function and by z ∈ M parameters constrained to live on the manifold.

Original: The vanilla gradient descent step in Euclidean space is given by:

Original: [Equation]

Original: In Riemannian optimization Bonnabel (2013), one replaces this update scheme by

Original: [Equation]

Original: Note that for M = R n , we recover again the vanilla gradient descent step.

Original: If calculating the exponential map is not possible or feasible, it is replaced by the aforementioned retraction R z .

Original: For intuition, the Riemannian gradient ∇ M z indicates the best direction in the tangent space T z M

Original: and we map the resulting direction in the tangent space down to the manifold, ensuring z ∈ M .

Original: B. A N E XAMPLE : T HE S PHERE S n − 1 R

Original: In order to provide further understanding and intuition for the introduced quantities, we will study the Riemannian manifold S n − 1 R

Original: [Equation]

Original: } equipped with the metric tensor g p =

Original: [Equation]

Original: inherited from the ambient space R n .

Original: For p ∈ S n − 1 R the tangent space at p is given by

Original: [Equation]

Original: This holds because for any smooth curve γ on S n − 1 , we have || γ ( t )

Original: ||

Original: 2 = γ ( t ) T γ ( t )

Original: =

Original: R , hence γ ( t ) T ˙ γ ( t ) = 0 and thus for t = 0 : p T v = 0 .

Original: Next consider the unit speed geodesics on S n − 1 R .

Original: Notice that although the metric tensor is the

Original: Euclidean one, the inﬁmum in (2.1) is taken over less curves γ since we are restricted to the sphere.

Original: In particular, straight lines are not included in the set of allowed curves.

Original: The minimum is achieved by so-called great circles , parametrized by

Original: [Equation]

Original: The induced distance on the sphere is given by

Original: [Equation]

Original: which corresponds to the arc length of the great circle connecting the points x, y ∈ S n − 1 .

Original: Using the equation for the unit speed geodesic, one easily arrives at the formula for the exponential map :

Original: [Equation]

Original: The logarithmic map is given by:

Original: [Equation]

Original: One important observation is the following.

Original: Intuitively, one would expect that we recover Euclidean space again when the radius R of the sphere goes to inﬁnity.

Original: This is due to the vanishing curvature c = 1 R 2 , since space starts to feel more ﬂat the bigger the sphere gets.

Original: The problem is that the formulas for the distance, exponential map and all other quantities do not recover their Euclidean counterpart .

Original: This follows from the chosen parametrization, which lets the point on the manifold implicitly depend on the curvature c through the hard constraint

Original: || x

Original: || = 1 √ c , making a recovery impossible.

Original: Convergence to Euclidean space will be important for the machine learning algorithms studied in this paper and as a result, we will work with a different isometric model of the sphere S n − 1 .

Original: C G RAPH E MBEDDINGS IN N ON -E UCLIDEAN G EOMETRIES
Revised: B G RAPH E MBEDDINGS IN N ON -E UCLIDEAN G EOMETRIES

Original: In this section we will motivate non-Euclidean embeddings of graphs and show why the underlying geometry of the embedding space can be very beneﬁcial for its representation.
Revised: In this section we will motivate non-Euclidean embeddings of graphs and show why the underlying geometry of the embedding space can be very beneﬁcial for its representation.

Original: We ﬁrst introduce a measure of how well a graph is represented by some embedding f : V −→ X , i (cid:55)→ f
Revised: We ﬁrst introduce a measure of how well a graph is represented by some embedding f : V −→ X , i (cid:55)→ f

Original: ( i ) :
Revised: ( i ) :

Original: Deﬁnition C.1.
Revised: Deﬁnition 3.

Original: Given an embedding f : V −→ X , i (cid:55)→ f ( i ) of a graph G =
Revised: Given an embedding f : V −→ X , i (cid:55)→ f ( i ) of a graph G =

Original: ( V, A ) in some metric space X , we call f a D-embedding for D ≥ 1 if there exists r > 0 such that
Revised: ( V, A ) in some metric space X , we call f a D-embedding for D ≥ 1 if there exists r > 0 such that

Original: [Equation]
Revised: [Equation]

Original: The inﬁmum over all such D is called the distortion of f .
Revised: The inﬁmum over all such D is called the distortion of f .

Original: The r in the deﬁnition of distortion allows for scaling of all distances.
Revised: The r in the deﬁnition of distortion allows for scaling of all distances.

Original: Note further that a perfect embedding is achieved when D = 1 .
Revised: Note further that a perfect embedding is achieved when D = 1 .

Original: C. T REES AND H YPERBOLIC S PACE
Revised: B. T REES AND H YPERBOLIC S PACE

Original: Trees are graphs that do not allow for a cycle, in other words there is no node i ∈ V for which there exists a path starting from i and returning back to i without passing through any node twice.
Revised: Trees are graphs that do not allow for a cycle, in other words there is no node i ∈ V for which there exists a path starting from i and returning back to i without passing through any node twice.

Original: The number of nodes increases exponentially with the depth of the tree.
Revised: The number of nodes increases exponentially with the depth of the tree.

Original: This is a property that prohibits
Revised: This is a property that prohibits

Original: Euclidean space from representing a tree accurately.
Revised: Euclidean space from representing a tree accurately.

Original: What intuitively happens is that "we run out of space".
Revised: What intuitively happens is that "we run out of space".

Original: Consider the trees depicted in ﬁg. 7.
Revised: Consider the trees depicted in ﬁg. 5.

Original: Here the yellow nodes represent the roots of each tree.
Revised: Here the yellow nodes represent the roots of each tree.

Original: Notice how rapidly we struggle to ﬁnd appropriate places for nodes in the embedding space because their number increases just too fast.
Revised: Notice how rapidly we struggle to ﬁnd appropriate places for nodes in the embedding space because their number increases just too fast.

Original: Moreover, graph distances get extremely distorted towards the leaves of the tree.
Revised: Moreover, graph distances get extremely distorted towards the leaves of the tree.

Original: Take for instance the green and the pink node.
Revised: Take for instance the green and the pink node.

Original: In graph distance they are very far apart as one has to travel up all the way to the root node and back to the border.
Revised: In graph distance they are very far apart as one has to travel up all the way to the root node and back to the border.

Original: In Euclidean space however, they are very closely embedded in a L 2 -sense, hence introducing a big error in the embedding.
Revised: In Euclidean space however, they are very closely embedded in a L 2 -sense, hence introducing a big error in the embedding.

Original: This problem can be very nicely illustrated by the following theorem:
Revised: This problem can be very nicely illustrated by the following theorem:

Original: Theorem 2.
Revised: Theorem 7.

Original: Consider the tree K 1 , 3 (also called 3-star) consisting of a root node with three children.
Revised: Consider the tree K 1 , 3 (also called 3-star) consisting of a root node with three children.

Original: Then every embedding { x 1 , . . .
Revised: Then every embedding { x 1 , . . .

Original: , x 4 } with x i ∈ R
Revised: , x 4 } with x i ∈ R

Original: k achieves at least distortion 2 √ 3 for any k ∈ N .
Revised: k achieves at least distortion 2 √ 3 for any k ∈ N .

Original: Proof.
Revised: Proof.

Original: We will prove this statement by using a special case of the so called Poincaré-type inequalities Deza & Laurent (1996):
Revised: We will prove this statement by using a special case of the so called Poincaré-type inequalities (Deza & Laurent, 1996):

Original: For any b 1 , . . .
Revised: For any b 1 , . . .

Original: , b k ∈ R with (cid:80) ki =1 b i = 0 and points x 1 , . . . , x k ∈ R n it holds thatk (cid:88) i,j = b i b j || x
Revised: , b k ∈ R with (cid:80) ki =1 b i = 0 and points x 1 , . . . , x k ∈ R n it holds that

Original: i − x j || 2 ≤
Revised: [Equation]

Original: Consider now an embedding of the tree x 1 , . .
Revised: Consider now an embedding of the tree x 1 , . .

Original: , x 4 where x 1 represents the root node.
Revised: , x 4 where x 1 represents the root node.

Original: Choosing b 1 = − 3 and b i = 1 for i (cid:54) = 1 leads to the inequality
Revised: Choosing b 1 = − 3 and b i = 1 for i (cid:54) = 1 leads to the inequality

Original: [Equation]
Revised: [Equation]

Original: The left-hand side of this inequality in terms of the graph distance is
Revised: The left-hand side of this inequality in terms of the graph distance is

Original: [Equation]
Revised: [Equation]

Original: and the right-hand side is
Revised: and the right-hand side is

Original: [Equation]
Revised: [Equation]

Original: As a result, we always have that the distortion is lower-bounded by 
Revised: As a result, we always have that the distortion is lower-bounded by (cid:113) 129 = 2 √

Original: [Equation] Euclidean space thus already fails to capture the geometric structure of a very simple tree.
Revised: Euclidean space thus already fails to capture the geometric structure of a very simple tree.

Original: This problem can be remedied by replacing the underlying Euclidean space by hyperbolic space.
Revised: This problem can be remedied by replacing the underlying Euclidean space by hyperbolic space.

Original: Consider again the distance function in the Poincaré model, for simplicity with c = 1 :
Revised: Consider again the distance function in the Poincaré model, for simplicity with c = 1 :

Original: [Equation]
Revised: [Equation]

Original: Assume that the tree is embedded in the same way as in ﬁg. 7, just restricted to lie in the disk of radius 1 √ c = 1 .
Revised: Assume that the tree is embedded in the same way as in ﬁg. 5, just restricted to lie in the disk of radius 1 √ c = 1 .

Original: Notice that as soon as points move closer to the boundary ( || x || −→ 1 ), the fraction explodes and the resulting distance goes to inﬁnity.
Revised: Notice that as soon as points move closer to the boundary ( || x || −→ 1 ), the fraction explodes and the resulting distance goes to inﬁnity.

Original: As a result, the further you move points to the border, the more their distance increases, exactly as nodes on different branches are more distant to each other the further down they are in the tree.
Revised: As a result, the further you move points to the border, the more their distance increases, exactly as nodes on different branches are more distant to each other the further down they are in the tree.

Original: We can express this advantage in geometry in terms of distortion:
Revised: We can express this advantage in geometry in terms of distortion:

Original: Theorem 3.
Revised: Theorem 8.

Original: There exists an embedding x 1 , . . .
Revised: There exists an embedding x 1 , . . .

Original: , x 4 ∈ P 2 for K 1 , 3 achieving distortion 1 + (cid:15) for (cid:15) > 0 arbitrary small.
Revised: , x 4 ∈ P 2 for K 1 , 3 achieving distortion 1 + (cid:15) for (cid:15) > 0 arbitrary small.

Original: Proof.
Revised: Proof.

Original: Since the Poincaré distance is invariant under Möbius translations we can again assume that x 1 = 0 .
Revised: Since the Poincaré distance is invariant under Möbius translations we can again assume that x 1 = 0 .

Original: Let us place the other nodes on a circle of radius r .
Revised: Let us place the other nodes on a circle of radius r .

Original: Their distance to the root is now given as
Revised: Their distance to the root is now given as

Original: [Equation]
Revised: [Equation]

Original: By invariance of the distance under centered rotations we can assume w.l.o.g. x
Revised: By invariance of the distance under centered rotations we can assume w.l.o.g. x

Original: 2 = ( r, 0) .
Revised: 2 = ( r, 0) .

Original: We further embed
Revised: We further embed

Original: • x 3 = (cid:0) r cos( 23 π ) , r sin( 23 π ) (cid:1) =
Revised: • x 3 = (cid:0) r cos( 23 π ) , r sin( 23 π ) (cid:1) =

Original: (cid:16) − r 2 , √ 32 r (cid:17) • x 4 = (cid:0) r cos( 4 3 π ) , r sin( 4 3 π ) (cid:1)
Revised: (cid:16) − r 2 , √ 32 r (cid:17) • x 4 = (cid:0) r cos( 43 π ) , r sin( 43 π ) (cid:1)

Original: = (cid:16) − r 2 , √ 3 2 r (cid:17) .
Revised: = (cid:16) − r 2 , √ 32 r (cid:17) .

Original: This procedure gives:
Revised: This procedure gives:

Original: [Equation]
Revised: [Equation]

Original: If we let the points now move to the border of the disk we observe that
Revised: If we let the points now move to the border of the disk we observe that

Original: [Equation]
Revised: [Equation]

Original: But this means in turn that we can achieve distortion 1 + (cid:15) for (cid:15) > 0 arbitrary small.
Revised: But this means in turn that we can achieve distortion 1 + (cid:15) for (cid:15) > 0 arbitrary small.

Original: QED.
Revised: QED.

Original: The tree-likeliness of hyperbolic space has been investigated on a deeper mathematical level.
Revised: The tree-likeliness of hyperbolic space has been investigated on a deeper mathematical level.

Original: Sarkar (2011) show that a similar statement as in Theorem 2 holds for a very general class of trees.
Revised: (Sarkar, 2011) show that a similar statement as in theorem 8 holds for all weighted or unweighted trees.

Original: The interested reader is referred to Hamann (2017); Sarkar (2011) for a more in-depth treatment of the subject.
Revised: The interested reader is referred to (Hamann, 2017; Sarkar, 2011) for a more in-depth treatment of the subject.

Original: Cycles are the subclasses of graphs that are not allowed in a tree.
Revised: Cycles are the subclasses of graphs that are not allowed in a tree.

Original: They consist of one path that reconnects the ﬁrst and the last node: ( v 1 , . .
Revised: They consist of one path that reconnects the ﬁrst and the last node: ( v 1 , . .

Original: . , v n , v 1 ) .
Revised: . , v n , v 1 ) .

Original: Again there is a very simple example of a cycle, hinting at the limits Euclidean space incurs when trying to preserve the geometry of these objects Matousek (2013).
Revised: Again there is a very simple example of a cycle, hinting at the limits Euclidean space incurs when trying to preserve the geometry of these objects (Matousek, 2013).

Original: Theorem 4.
Revised: Theorem 9.

Original: Consider the cycle G =
Revised: Consider the cycle G =

Original: ( V, E ) of length four.
Revised: ( V, E ) of length four.

Original: Then any embedding ( x 1 , . . .
Revised: Then any embedding ( x 1 , . . .

Original: , x 4 ) where x i ∈
Revised: , x 4 ) where x i ∈

Original: R
Revised: R

Original: k achieves at least distortion √ 2 .
Revised: k achieves at least distortion √ 2 .

Original: Proof.
Revised: Proof.

Original: Denote by x 1 , x 2 , x 3 , x 4 the embeddings in Euclidean space where x 1 , x 3 and x 2 , x 4 are the pairs without an edge.
Revised: Denote by x 1 , x 2 , x 3 , x 4 the embeddings in Euclidean space where x 1 , x 3 and x 2 , x 4 are the pairs without an edge.

Original: Again using the Poincaré-type inequality with b 1 = b 3 = 1 and b 2 = b 4 = −leads to the short diagonal theorem [-Matousek (2013): || x 1 − x 3 || 
 + || x 2 − x 4 || 2 ≤ || x 1 − x 2 || 
-]
Revised: Again using the Poincaré-type inequality with b 1 = b 3 = 1 and b 2 = b 4 = −leads to the short diagonal theorem [-Matousek (2013): || x 1 − x 3 || 
 + || x 2 − x 4 || 2 ≤ || x 1 − x 2 || 
-](Matousek, 2013):

Original: [-+ || x 2 − x 3 || 
 + || x 3 − x 4 || + || x 4 − x 1 ||-]
Revised: [-+ || x 2 − x 3 || 
 + || x 3 − x 4 || + || x 4 − x 1 ||-][Equation]

Original: The left hand side of this inequality in terms of the graph distance is[-d G (1 , 3) 2 + d G (2 , 4) 2 = 2 
-]
Revised: The left hand side of this inequality in terms of the graph distance is[-d G (1 , 3) 2 + d G (2 , 4) 2 = 2 
-]

Original: [-+ 2 2 = 
 and the right hand side is 1 2 + 1 2 + 1 2 + 1 2 = 4 .-] Therefore any embedding has to shorten one diagonal by at least a factor √ 2 .
Revised: [-+ 2 2 = 
 and the right hand side is 1 2 + 1 2 + 1 2 + 1 2 = 4 .-] Therefore any embedding has to shorten one diagonal by at least a factor √ 2 .

Original: It turns out that in spherical space, this problem can be solved perfectly in one dimension for any cycle.
Revised: It turns out that in spherical space, this problem can be solved perfectly in one dimension for any cycle.

Original: Theorem 5.
Revised: Theorem 10.

Original: Given a cycle G =
Revised: Given a cycle G =

Original: ( V, E ) of length n , there exists an embedding { x 1 , . . .
Revised: ( V, E ) of length n , there exists an embedding { x 1 , . . .

Original: , x n } achieving distortion 1.
Revised: , x n } achieving distortion 1.

Original: Proof.
Revised: Proof.

Original: We model the one dimension spherical space as the circle S 1 .
Revised: We model the one dimension spherical space as the circle S 1 .

Original: Placing the points at angles 2 πin and using the arclength on the circle as the distance measure leads to an embedding of distortion 1 as all pairwise distances are perfectly preserved.
Revised: Placing the points at angles 2 πin and using the arclength on the circle as the distance measure leads to an embedding of distortion 1 as all pairwise distances are perfectly preserved.

Original: Notice that we could also use the exact same embedding in the two dimensional stereographic projection model with c = 1 and we would also obtain distortion 1.
Revised: Notice that we could also use the exact same embedding in the two dimensional stereographic projection model with c = 1 and we would also obtain distortion 1.

Original: The difference to the Poincaré disk is that spherical space is ﬁnite and the border does not correspond to inﬁnitely distant points.
Revised: The difference to the Poincaré disk is that spherical space is ﬁnite and the border does not correspond to inﬁnitely distant points.

Original: We therefore have no (cid:15) since we do not have to pass to a limit.D G YROVECTOR S PACES
Revised: We therefore have no (cid:15) since we do not have to pass to a limit.

Original: Euclidean space is so popular because it allows for a lot of structure.

Original: We have well-deﬁned operations such as addition of vectors or scalar multiplication between a real number and a vector, which obey certain rules.

Original: Those operations are the building blocks for more sophisticated concepts like matrix multiplication, which in turn are the building blocks for many machine learning models.

Original: In order to have a principled extension of the algorithms of interest, we need a translation of these quantities to the non-Euclidean setting.

Original: Gyrovector spaces Ungar (2008) offer such a generalization and will be studied in the following section.

Original: D. V ECTOR S PACES AND  G YROVECTOR S PACES
Revised: C S PHERICAL S PACE AND ITS G YROSTRUCTURE

Original: We wish to extend the notion of Euclidean vector spaces.

Original: For the Euclidean space R d , the addition and scalar multiplication are given by the well-known component-wise vector addition and scalar multiplication.

Original: The inner product is given by the usual dot product x T y .

Original: It is clear that we have to give up on some part of the structure, as any real vector space of dimension d is isomorphic to R d .

Original: In order to answer how much we can retain, we ﬁrst need to introduce a mathematical object residing on a deeper level, namely the equivalent to a group.

Original: This dual group will be the foundation for the extension of a vector space.

Original: It turns out that we already have to give up on some of the structure on the group level, as can be seen by the deﬁnition of a so called gyrogroup Ungar (2008):

Original: Deﬁnition D.1. A gyrogroup is a set G withwell-deﬁned binary operation ⊕ satisfying the following axioms: There exists at least one 0 ∈ G such that for every g ∈ G : 0 ⊕ g = g 2.
Revised: Contrarily to hyperbolic geometry, spherical geometry is not only in violation with the ﬁfth postulate of Euclid but also with the ﬁrst.

Original: For every g ∈ G there exists (cid:9) g ∈ G such that (cid:9) g ⊕ g =3.

Original: For any a, b, c ∈ G there exists a unique element gyr [ a, b ] c ∈ G such that a ⊕ ( b ⊕ c ) =

Original: ( a ⊕ b ) ⊕ gyr [ a, b ] c 4.

Original: The map c −→ gyr

Original: [ a, b ] c is an automorphism satisfying gyr

Original: [ a, b ] = gyr [ a ⊕ b, b ] We see that we lose the associative law and replace it by the weaker notion stated in 3 and 4. Equipped with this deﬁnition, we introduce the concept of a real gyrovector space Ungar (2008):
Revised: Notice that, shortest paths are not unique as for antipodal (oppositely situated) points, we have inﬁnitely many geodesics connecting the two.

Original: Deﬁnition D.2.

Original: A real inner product gyrovector space is the triple ( G, ⊕ , ⊗ ) where ( G, ⊕ ) is a gyrocommutative gyrogroup fullﬁlling the following axioms:

Original: G is a subset of a real inner product space V , inheriting the inner product (cid:104) x, y (cid:105) which is invariant under gyrations: (cid:104) gyr

Original: [ x, y ] a, gyr [ x, y ] b (cid:105) =

Original: (cid:104) x, y (cid:105) 2. There exists 1 ∈ G such that 1 ⊗ x = x for any x ∈ G (Identity) 3. ( s 1 + s 2 ) ⊗ x = s 1 ⊗ x ⊕ s 2 ⊗ x (Scalar Distributive Law) 4.
Revised: Hence the ﬁrst axiom does not hold.

Original: ( s 1 s 2 ) ⊗ x = s 1 ⊗ ( s 2 ⊗ x ) (Scalar Associativity) 5.

Original: | s |⊗ x || s ⊗ x || = x || x || (Scaling) 6. gyr [ x, y ]( s ⊗ z )
Revised: Notice that the third postulate holds as we stated it

Original: = s ⊗ gyr

Original: [ x, y ]

Original: z 7.

Original: gyr [ s 1 ⊗ x, s 2 ⊗ x ] = id 8.

Original: ||

Original: G ||

Original: = { ± || a || : a ∈ G } with ⊕ and ⊗ forms a vector space. || r ⊗ x || = | r | ⊗ || x || (Homogeneity) 10.
Revised: but it is sometimes also phrased as: "A circle of any center and radius can be constructed".

Original: || x ⊕ y

Original: ||

Original: ≤

Original: || x || ⊕

Original: ||

Original: y ||

Original: (Gyrotriangle inequality)

Original: Note that we deﬁne ⊕ and ⊗ for the norms of a ∈ G as seen in 9 and 10 in perfect analogy to the vectorized version, by treating the scalars as one dimensional vectors.

Original: In the following sections we will study two particular examples of manifolds that give rise to a gyrostructure, namely hyperbolic and spherical space.

Original: Together with Euclidean space, they are the only manifolds inducing a geometry with constant sectional curvature.

Original: D. H YPERBOLIC S PACE AND ITS G YROSTRUCTURE

Original: There are several isometric models of hyperbolic geometry.

Original: Each model offers some advantages and disadvantages in its parametrization.

Original: It is important to notice that these advantages and disadvantages only arise through the chosen parametrizations , all the models being isometric.

Original: In this work, we chose to work with the Poincaré model .

Original: Its conformality property and the pioneering work in this model by Nickel & Kiela (2017); Ganea et al.

Original: (2018a;b) facilitated the extension of the algorithms of consideration to the non-Euclidean setting.

Original: D.2. G YROVECTOR S PACE IN THE P OINCARÉ B ALL

Original: This section is based on the extensive research of Ungar (2008) on hyperbolic space and its underlying gyrovector structure, as well as Ganea et al.

Original: (2018b) who connected gyro theory and machine learning in their work.

Original: For proofs of the introduced results, we hence refer to the aforementioned sources.

Original: Inspired by general relativity and relativistic velocity addition , Ungar (2008) discovered the notion of gyro theory.

Original: General relativity and hyperbolic space are very tightly related and it turns out that the relativistic velocity addition is exactly the gyro addition ⊕ E in the Klein model:

Original: [Equation]

Original: This addition can be translated to the Poincaré ball, leading to the so called Möbius addition: Deﬁnition D.3. For x, y ∈ P dc , the Möbius addition is given by: [Equation]
Revised: Due to the ﬁniteness of space we cannot have arbitrary large circles and hence phrased that way, the third postulate would not hold.

Original: Notice that as desired, x ⊕ c M y c −→ 0 −−−→ x + y recovering Euclidean addition in the limit. We can further deﬁne a scalar multiplication: Deﬁnition D.4.
Revised: Finally, we replace the ﬁfth postulate by:

Original: For s ∈ R and x ∈ ⊗ c M , the scalar multiplication in the Poincaré model is given by: [Equation] Again observe that in the limit s ⊗ c M x c −→ 0 −−−→ sx .One can show that Möbius addition and scalar multiplication in P dc form a gyrovector space Ungar (2008) as introduced in Deﬁnition D.2.
Revised: • Given any straight line l and a point p not on l , there exists no shortest line g passing through p but never intersecting l .

Original: We will now list various results, illustrating how the role of Möbius addition and scalar multiplication mimics the one observed in Euclidean geometry with component-wise addition and multiplication.

Original: We can use the Möbius addition to rewrite the distance formula:

Original: Lemma 3.

Original: [Equation] Notice how the distance depends  on x and y only through || −  x⊕ c M y || ,as is the case for Euclidean geometry where ⊕ c M becomes + . This model is smoothly deformed to the Euclidean model in the curvature limit as d c P ( x, y ) c −→ 0 −−−→ 2 (cid:107) x − y (cid:107) .
Revised: The standard model of spherical geometry suffers from the fact that its underlying space depends directly on the curvature κ through a hard constraint − κ (cid:104) x , x (cid:105) =

Original: Moreover, the introduced operations interact naturally with the underlying geometry, as can be seen by the following properties Ungar (2008):

Original: Corollary 5.1.

Original: For x, y, z ∈ P dc and n ∈ N , it holds

Original: : • d c P ( x ⊕ c M z, y ⊕ c M z ) = d c P ( x, y ) • || r ⊗ c M x || P = r || x || P where || x || P =
Revised: 1 (similarly to the Lorentz model of hyperbolic geometry).

Original: d c P ( x, 0) • n ⊗ c M x = x ⊕ c M . . . ⊕ c M x ,where the addition is performed n times. One can see that Möbius translations leave the distance invariant and Möbius scaling on the other hand transforms the norm || x || P appropriately, in perfect duality to the Euclidean world.
Revised: Indeed, when κ → 0 , the domain diverges to a sphere of inﬁnite radius which is not well deﬁned.

Original: We can also write the formula for geodesics in the Poincaré model in an intuitive manner Ungar (2008): Lemma 4. For x, y ∈ P dc , the geodesic γ x −→ y : R −→ P dc , connecting x and y is given by [Equation]
Revised: For hyperbolic geometry, we could circumvent the problem by moving to the Poincaré model, which is the stereographic projection of the Lorentz model, relaxing the hard constraint to an inequality.

Original: This is in direct analogy to the Euclidean geodesics parametrized by γ x −→ y ( t ) =

Original: x + t ( y − x ) , where one simply replaces all Möbius operations by the Euclidean ones.

Original: As a consequence of this formula, we recover again the Euclidean geodesics for c −→ 0 . Finally, we also obtain the exponential and logarithmic map in closed form Ganea et al. (2018b): Lemma 5.
Revised: A similar solution is also possible for the spherical model.

Original: For x ∈ P dc and v (cid:54) = 0 , the exponential map exp x : T x P

Original: dc −→ P dc is given by

Original: [Equation]

Original: Moreover for y (cid:54) = x , log x : P dc −→ T

Original: x

Original: P dc can be expressed via

Original: [Equation]

Original: Observe that we obtain the Euclidean counterparts in the limit: exp cx ( v ) c −→ 0 −−−→ x + v and log

Original: [Equation]

Original: ( v ) [Equation] y − x . D. S PHERICAL S PACE AND ITS G YROSTRUCTURE
Revised: C. S TEREOGRAPHIC P ROJECTION M ODEL OF THE S PHERE

Original: We would like to work in a model of constant positive curvature that would recover the Euclidean space operations when the curvature goes to 0.

Original: Unfortunately, the parameterization of standard spherical geometry given by S dc = { x ∈ R d +1 | c (cid:107) x (cid:107) 2 = 1 } does not give achieve this goal.

Original: As a consequence, we will work in a different spherical model (still isometric with the standard one).

Original: D.3. S TEREOGRAPHIC P ROJECTION M ODEL OF THE S PHERE

Original: In the following we construct a model by isometrically projecting the S dc to R d (in perfect duality to the construction of the Poincaré model which is the isometric projection of the Lorentz model).
Revised: In the following we construct a model in perfect duality to the construction of the Poincaré model.

Original: Fix the south pole z
Revised: Fix the south pole z

Original: = (0 , − 1 √ c )  .
Revised: = (0 , − 1 √ κ ) of the sphere of curvature κ > 0 , i.e. of radius R := κ − 12 .

Original: The stereographic projection isthen the map:
Revised: The stereographic projection is the map:

Original: [Equation]
Revised: [Equation]

Original: with the inverse given by
Revised: with the inverse given by

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne η x =2 1+ c
Revised: where we deﬁne λ κ x =

Revised: [Equation]

Original:  || x
Revised: + κ || x

Original: || 2 .
Revised: || 2 .

Original: Again we take the image of the sphere S dR under the extended projection Φ((0 , . . .
Revised: Again we take the image of the sphere S nR under the extended projection Φ((0 , . . .

Original: , 0 , − 1 c ))
Revised: , 0 , − 1 κ ))

Original: = 0 , leading to the stereographic model of the sphere.
Revised: = 0 , leading to the stereographic model of the sphere.

Original: The metric tensor transforms as Janson (2015):
Revised: The metric tensor transforms as:

Original: [Equation]
Revised: [Equation]

Original: Notice that the metric tensor of the Poincaré model and the stereographic model agree exactly if written in terms of the curvature K and not its absolute value c .
Revised: Using Cauchy-Schwarz’s inequality, we have A := 1 − 2 κ x T y + κ 2 || x

Original: The metric tensor in turn induces the distance between x, y ∈ R d Janson (2015):
Revised: || 2 || y || 2 ≥ 1 − 2 | κ |(cid:107) x (cid:107)(cid:107) y (cid:107) + κ 2 || x

Original: [Equation]
Revised: || 2 ||

Original: This model is smoothly deformed to the Euclidean model in the curvature limit as d c G ( x, y ){+|| 
+}
Revised: y {+|| 
+}

Original: c −→ 0 −−−→ 2 (cid:107) x −  y (cid:107)  .
Revised: = (1 − | κ |(cid:107) x (cid:107)(cid:107) y (cid:107) ) 2 ≥ 0 .

Original: Moreover, replacing c (cid:55)→ − c with the identiﬁcation √− c = i √ c andthe application of Euler’s formula e ix = cos( x ) + i sin( x  ) , recovers exactly the Poincaré distance.
Revised: Since equality in the Cauchy-Schwarz inequality is only reached for colinear vectors, we have that A = 0 is equivalent to κ > 0 and x = y / ( κ (cid:107) y (cid:107) 2 ) .

Original: D.3. G YROVECTOR S PACE IN THE S TEREOGRAPHIC M ODEL
Revised: C.2. P ROOF OF T HEOREM

Original: As in the Poincaré model, we can again deﬁne an addition in the stereographic model:
Revised: Let us start by proving that for x ∈ R n and v ∈ T x R n the exponential map is given by exp [Equation] Indeed, take a unit speed geodesic γ x , v ( t ) starting from x with direction v . Notice that the unit speed geodesic on the sphere starting from x (cid:48) ∈ S n − 1 is given by Γ x (cid:48) , v (cid:48) ( t ) =

Original: Deﬁnition D.5.
Revised: x (cid:48) cos κ ( t ) + 1 √ κ sin κ ( t ) v (cid:48) . By the Egregium theorem, we know that Φ( γ x , v ( t )) is again a unit speed geodesic in the sphere where Φ − 1 : x (cid:55)→ x (cid:48) = (cid:16) λ κ x x , 1 √ κ ( λ κ x − 1) (cid:17) .

Original: For x, y ∈ R d , the stereographic addition is given by:
Revised: Hence Φ( γ x , v ( t )) is of the form of Γ for some x (cid:48) and v (cid:48) . We can determine those by [Equation] Notice that ∇ x λ κ x = − κ ( λ κ x ) 2 x and we thus get

Original: [Equation]
Revised: [Equation]

Original: Notice that as desired, x⊕ c G y c
Revised: We can obtain γ x , v again by inverting back by calculating γ x , v ( t ) [Equation] [Equation] x

Original: −→ 0 −−−→ x + y , recovering Euclidean addition.
Revised: [Equation] (cid:48) ( t )) , resulting in [Equation] Denoting g κ x ( v , v ) =

Revised: ||

Revised: v || 2 λ κ x we have that exp κ x ( v ) =

Revised: γ x ,

Revised: [Equation]

Revised: x

Revised: [Equation]

Original: Moreover, replacing c (cid:55)→ − c recovers Möbius addition.
Revised: v , [Equation] (cid:16)(cid:112) g κ x ( v , v ) (cid:17) which concludes the proof of the above formula of the exponential map. One then notices that it can be re-written in terms of the κ -addition.

Revised: The formula for the logarithmic map is easily checked by verifying that it is indeed the inverse of the exponential map.

Revised: Finally, the distance formula is obtained via the well-known identity d κ ( x , y )

Revised: = (cid:107) log κ x ( y ) (cid:107) x where (cid:107) v (cid:107) x =

Revised: (cid:112) g κ x ( v , v ) .

Revised: Note that as expected, exp κ x ( v ) → κ → 0 x + v , converging to the Euclidean exponential map.

Revised: (cid:3)

Revised: C.2. P ROOF OF T HEOREM

Revised: We ﬁrst compute a Taylor development of the κ -addition w.r.t κ around zero:

Revised: [Equation]

Revised: We then notice that using the Taylor of (cid:107) · (cid:107) 2 , given by (cid:107) x + v (cid:107) 2 = (cid:107) x (cid:107) 2 + (cid:104) x , v (cid:105)

Revised: + O ( (cid:107) v (cid:107) 22 ) for v → 0 , we get

Revised: [Equation]

Revised: [Equation]

Original: Actually, one can write both additions in a uniﬁed function of the sectional curvature K as
Revised: Finally Taylor developments of tan κ ( | κ | 12 u ) and | κ | − 12 tan − 1 κ ( u ) w.r.t κ around 0 for ﬁxed u yield For κ → 0 + , [Equation] For κ → 0 − ,

Original: [Equation]
Revised: [Equation]

Original: Inserting K = − c leads to Möbius addition, whereas K = c results in the steregraphic addition. We hence have an addition that interpolates smoothly between all three geometries.
Revised: [Equation] [Equation] The left and right derivatives match, hence even though κ (cid:55)→ | κ | 12 is not differentiable at κ = 0 , the function κ (cid:55)→ tan κ ( | κ | 12 u ) is.

Original: Moreover we can deﬁne a scalar multiplication:
Revised: A similar analysis yields the same conclusion for κ (cid:55)→ [Equation]

Original: Deﬁnition D.6.
Revised: yielding For

Original: For s ∈ R and x ∈ R d , the scalar multiplication in the stereographic model is given by:
Revised: [Equation] → [Equation] Since a composition of differentiable functions is differentiable

Original: [Equation]
Revised: [Equation]

Original: Notice that we have a perfect correspondence as well by using Euler’s identity and  the formulas tan( ix ) = i tanh( x ) and tan − 1 ( ix )
Revised: we consequently obtain that ⊗ κ , exp κ , log κ and d κ are differentiable functions of κ , under the assumptions on x , y , v stated in

Original: =
Revised: Theorem 3. Finally, the Taylor development of d κ follows by composition of Taylor developments: [Equation] C.2. P ROOF OF T HEOREM

Original:  itanh − 1 ( x ) . Furthermore, we state the following theorem:
Revised: If A = I n then for all i we have (cid:80) j A ij =

Original: Theorem 6.
Revised: [Equation]

Original: The triple ( R d , ⊕ c G , ⊗ c G ) equipped with the metric tensor g x = η 2 x I deﬁnes a gyrovector space.
Revised: hence [Equation] For associativity, we ﬁrst note that the gyromidpoint is unchanged by a scalar rescaling of A . The property then follows by scalar associativity of the κ -scaling.

Original: Proof is by algebraic veriﬁcation of the identities in deﬁnition D.2.
Revised: [Equation] C.2. P ROOF OF T HEOREM It is proved in (Ungar, 2005) that the gyromidpoint commutes with isometries. The exact same proof holds for positive curvature, with the same algebraic manipulations.

Original: In the following, we will introduce the statements dual to the ones in  the Poincaré ball and obtain formulas for the exponential and logarithmic map.
Revised: Moreover, when the matrix A is right-stochastic, for each row, the sum over columns gives 1, hence our operation (cid:2) κ reduces to a gyromidpoint. As a consequence, our (cid:2) κ commutes with isometries in this case. Since isometries preserve distance, we have proved the theorem. [Equation]

Original: As in Lemma 3, we can rewrite the distance formula more elegantly:
Revised: We begin our proof by stating the left-cancellation law : [Equation] and the following simple identity stating that orthogonal maps commute with κ -addition

Original: Lemma 6.
Revised: [Equation]

Original: For x, y ∈ R d ,  the distance d c G ( x, y ) can be written as:
Revised: Next, we generalize the gyro operator from Möbius gyrovector spaces as deﬁned in Ungar (2008): [Equation] Note that this deﬁnition applies only for u , v , w ∈ st dκ for which the κ -addition is deﬁned (see theorem 1). Following Ungar (2008), we have an alternative formulation (veriﬁable via computer algebra):

Original: [Equation]
Revised: [Equation]

Original: Corollary 6.1. For x, y, z ∈ R d and n ∈ N , it holds: • d c G ( z ⊕ c G x, z ⊕ c G y )
Revised: where the quantities A, B, D have the following closed-form expressions:

Original: =
Revised: [Equation]

Original: d c G ( x, y ) • || r ⊗ c G x || G = | r ||| x
Revised: We then have the following relations:

Original: || G where
Revised: Lemma 11.

Original: || x || G = d c G ( x, 0) • n ⊗ c
Revised: For all u , v , w ∈ st dκ for which the κ -addition is deﬁned we have the following relations: i) gyration is a linear map, ii) u

Original: G x = x ⊕ c G . . .
Revised: ⊕ κ v = gyr

Original:  ⊕ c
Revised: [ u , v ]( v ⊕ κ u ) , iii) − ( z ⊕ κ u ) ⊕ κ ( z ⊕ κ v ) = gyr [ z , u ]( − u ⊕ κ v ) , iv) (cid:107) gyr [ u , v ] w (cid:107) = (cid:107) w (cid:107) .

Original: G x where the addition is performed n times.
Revised: Proof. The proof is similar with the one for negative curvature given in Ungar (2008). The fact that gyration is a linear map can be easily veriﬁed from its deﬁnition. For the second part, we have − gyr [ u , v ]

Original: Notice how Corollary 6.1 is  the exact analog to Corollary 5.1, again showing how natural the deﬁned operations interact with the geometry.
Revised: [Equation] v [Equation] (44) where the ﬁrst equality is a trivial consequence of the fact that gyration is a linear map, while the last equality is the consequence of left-cancellation law.

Original: For the exponential map, we ﬁrst derive an expression using the Egregium theorem and the known formulas for unit speed geodesics on the sphere S dc .
Revised: The third part follows easily from the deﬁnition of the gyration and the left-cancellation law. The fourth part can be checked using the alternate form in eq. We now follow Ungar (2014) and describe all isometries of st dκ spaces: Theorem 12.

Original: This leads to a closed formula not involving any stereographic operations.
Revised: Any isometry φ of st dκ can be uniquely written as: [Equation] The proof is exactly the same as in theorems 3.19 and 3.20 of Ungar (2014), so we will skip it.

Original: We then proceed to rewrite this expression in terms of ⊕ c G , leading to a term that is easily invertible and hence providing us with the logarithmic map.
Revised: We can now prove the main theorem. Let φ ( x ) = z ⊕ κ Rx be any isometry of st d κ , where R ∈ O ( d ) is an orthogonal matrix.

Original: Corollary 6.2.
Revised: Let us denote by v := (cid:80) ni

Original: For{+=
 α i log κ+} x ∈ R d and v ∈ T x R d the exponential map can be recast as
Revised: {+=
 α i log κ+} x ( x i ) . Then, using lemma 11 and the formula of the log map from theorem 2, one obtains the following identity:

Original: [Equation]
Revised: [Equation]

Original: Moreover for y (cid:54) = x , log x : R d −→ T x R d the logarithmic map is given by
Revised: and, thus, using the formula of the exp map from theorem 2 we obtain:

Original: [Equation]
Revised: [Equation]

Original: As expected, exp cx ( v ) c
Revised: Using eq. (39), we get that

Original: −→ 0 −−−→ x + v , converging to the Euclidean exponential map.
Revised: [Equation]

Original: For the proof we again use the same arguments as in the proof of theorem 6. Once more, observe the perfect correspondence between the Poincaré (Lemma 5) and the stereographic models.
Revised: giving the desired [Equation]

Original: E H YPERBOLIC L OGITS
Revised: D L OGITS

Original: The ﬁnal element missing in the hyperbolic neural network is the logit layer, a neccessity for any classiﬁcation task.
Revised: The ﬁnal element missing in the κ -GCN is the logit layer, a necessity for any classiﬁcation task.

Original: We here use the formulation of Ganea et al. (2018b).
Revised: We here use the formulation of (Ganea et al., 2018a).

Original: Denote by { 1 , . .
Revised: Denote by { 1 , . .

Original: , K } the possible labels and let a k ∈ R d , b k ∈ R and x ∈ R d .
Revised: , K } the possible labels and let a k ∈ R d , b k ∈ R and x ∈ R d .

Original: The output of a feed forward neural network for classiﬁcation tasks is usually of the form
Revised: The output of a feed forward neural network for classiﬁcation tasks is usually of the form

Original: [Equation]
Revised: [Equation]

Original: b [Equation] In order to generalize this expression to hyperbolic space, the authors of Ganea et al. (2018b) realized that the term in the softmax can be rewritten as
Revised: In order to generalize this expression to hyperbolic space, the authors of (Ganea et al., 2018a) realized that the term in the softmax can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: where H a,b = { x ∈ R d : (cid:104) x, a (cid:105) − b  } = { x ∈ R d : (cid:104)− p + x, a (cid:105)} =
Revised: where H a ,b = { x ∈ R d : (cid:104) x , a (cid:105) − b = 0 } = { x ∈ R d : (cid:104)− p + x , a (cid:105) = 0 } =

Original: ˜ H a,p with p ∈ R d .
Revised: ˜ H a , p with p ∈ R d .

Original: As a ﬁrst step, they deﬁne the hyperbolic hyperplane as
Revised: As a ﬁrst step, they deﬁne the hyperbolic hyperplane as

Original: [Equation]
Revised: [Equation]

Original: where now a ∈ T p P dc and p ∈ P dc .
Revised: where now a ∈ T p st dκ and p ∈ st dκ .

Original: They then proceed with the following lemma: Lemma 7.
Revised: They then proceed proving the following formula:

Original: [Equation]
Revised: [Equation]

Original: Using this equation, they were able to obtain the following expression for the logit layer:Deﬁnition E.1. The logits in hyperbolic space are deﬁned as
Revised: Using this equation, they were able to obtain the following expression for the logit layer:

Original: [Equation]
Revised: [Equation]

Original: where a k ∈ T 0 P dc ∼ = R d , x ∈ P dc and p k ∈ P dc .
Revised: where a k ∈ T 0 st dκ ∼ = R d , x ∈ st dκ and p k ∈ st dκ .

Original: Combining all these operations leads to the deﬁnition of a hyperbolic feed forward neural network.
Revised: Combining all these operations leads to the deﬁnition of a hyperbolic feed forward neural network.

Original: Notice that the weight matrices W and the normal vectors a k live in Euclidean space and hence can be optimized by standard methods such asADAM Kingma & Ba (2015). F GCN S IN P RODUCT OF C ONSTANT C URVATURE S PACES
Revised: Notice that the weight matrices W and the normal vectors a k live in Euclidean space and hence can be optimized by standard methods such as

Original: Inspired by Gu et al.

Original: (2019), we can now take the model one step further and study graph neural networks in a product of spaces consisting of spherical and hyperbolic components.

Original: More precisely, we study spaces of the form

Original: [Equation]

Original: where we denote by M dc − the Poincaré representation of H dc − and by G bc + the stereographic representation of S bc + .

Original: For v ∈ X , write

Original: [Equation]

Original: where each v i ∈ M d i and ˜ v i ∈

Original: G b i represents a hyperbolic and spherical component.

Original: It can be shown that X is again a Riemannian manifold equipped with a distance function, an exponential map and a logarithmic map decomposing as follows Ficken (1939);

Original: Turaga & Srivastava (2016): Lemma 8.
Revised: ADAM (Kingma & Ba, 2015).

Original: For x, y ∈ X , the distance between the two points is given by:

Original: [Equation]

Original: Moreover for v ∈ T x X , the exponential map exp x : T x X −→ X decomposes as [Equation] Finally, fory ∈ X , the logarithmic map log x : X −→ T x X takes the form [Equation]
Revised: For positive curvature κ > 0 we use in our experiments the following formula for the softmax layer:

Original: One may ask why we are also splitting the hyperbolic and spherical components into products.

Original: After all, for Euclidean components it is a well-known fact that for any n, m ∈ N , we have that R n × R m =

Original: R n + m .

Original: This property does not hold anymore for both spherical and hyperbolic products of spaces.

Original: Consider for instance S 1 × S 1 .

Original: It turns out that this product of two circles is not isomorphic to the 2-sphere S 2 but rather two the torus T 2 .

Original: The reader may check this statement by calculating the fundamental groups of both S 1 × S 1 and S 2 and observe that they are not identical.

Original: Thus splitting both hyperbolic and spherical components leads to a more general class of models.

Original: F. G YROSTRUCTURE IN P RODUCTS OF G YROVECTOR SPACES

Original: It turns out that we can again equip X with a gyrovector space structure, inheriting the structure from all the smaller components:

Original: Theorem 7.

Original: For x, y ∈ X , deﬁne the addition ⊕ X and the scalar multiplication ⊗ X as:

Original: [Equation]
Revised: [Equation]

Original: • r ⊗

Original: X x = ( r ⊗ M d 1 x 1 , . .

Original: , r ⊗ M dm x m , r ⊗ G d 1 ˜ x 1 , . . .

Original: , r ⊗ G dl ˜ x m )

Original: Moreover for the norms

Original: || x ||

Original: and ||

Original: y

Original: ||

Original: where x, y ∈ X we deﬁne: • || x || ⊕
Revised: which is inspired from the formula i sin( x ) = sinh( ix ) where i :

Original: X

Original: ||

Original: y

Original: ||

Original: = (cid:113)(cid:80) mi

Original: =1 ( || x i ||

Original: ⊕ X

Original: i ||

Original: y

Original: [-i || ) 
-]
Revised: [-i || ) 
-]

Original: + (cid:80) li

Original: [-=
-]
Revised: [-=
-]=

Original: || ˜ x

Original: i || ⊕

Original: i X

Original: || ˜

Original: y i || )• r ⊗ X || x
Revised: √− 1 .

Original: || = (cid:113)(cid:80) mi

Original: =1 ( r ⊗ X

Original: i ||

Original: x i || )

Original: 2 + (cid:80) li =1 ( r ⊗ X

Original: i ||

Original: ˜ x

Original: i || )

Original: Then the triple ( X , ⊕ X , ⊗ X ) deﬁnes a gyrovector space.

Original: Moreover, we have the following two invariances:

Original: • || r ⊗ X x

Original: || X

Original: =

Original: d X ( r ⊗ X x, 0) =

Original: r || x

Original: || X • d X ( b ⊕ X x, b ⊕ X y ) =

Original: d X ( x, y )

Original: The gyrovector space over X allows us again to extend all the quantities derived in the previous chapter to the more general framework of products of Riemannian spaces of constant sectional curvature.

Original: F. R IGHT M ATRIX M ULTIPLICATION IN P RODUCT OF S PACES

Original: Given an embedding X ∈ R n × ( d 1 + ··· + d m + b 1 + ··· + b l ) where each row X i • ∈ X and a weight matrix

Original: W ∈ R ( d 1 + ··· + d m + b 1 + ··· + b l ) ×

Original: ( e 1 + ··· + e m + f 1 + ··· + f l ) , we want to deﬁne again the notion of a right matrix multiplication, leading to a mixing of the different features.

Original: First express the embedding through its different components:

Original: [Equation]

Original: X 11 • . . .

Original: X m 1 • ˜ X 11 • . . .

Original: ˜ X l 1 • ...

Original: X 1 n • . . .

Original: X mn • ˜

Original: X 11 • . . .

Original: ˜ X l

Original: [Equation]

Original: 

Original:  =

Original: [ X 1 , . .

Original: , X m , ˜ X 1 , . . .

Original: , ˜ X l ] where X ji • ∈ B d j c − j , ˜ x ji • ∈ B

Original: b j c + j , X i ∈ R n × d i and ˜ X i ∈ R n × b i .

Original: Moreover, express the weight matrix

Original: W as

Original: [Equation]

Original:  where W i ∈ R

Original: d i × ( e 1 + ··· + e m ) and ˜ W i ∈ R b i × ( f 1 + ··· + f l ) .

Original: Deﬁnition F.1.

Original: The right matrix multiplication in the product of Riemannian manifolds of constant sectional curvatures X is given by

Original: [Equation]

Original: Let us examine the behaviour of ⊗ X a bit more closely using the notation introduced above.

Original: We ﬁrst look at the inner term of the matrix multiplication:

Original: [Equation]

Original: Now the right multiplication with W can be expressed as the sum over the smaller matrices:

Original: [Equation]

Original: Notice that here a mixing of the different feature components is happening in the respective tangent spaces at zero.

Original: Hence this operation is not simply decomposing and the different components interact with each other.

Original: This interaction is then mapped back to the product manifold using the exponential map.

Original: Applications of non-linearities can also again be deﬁned using exponential and logarithmic maps:

Original: Deﬁnition F.2.

Original: Given any map σ : R d 1 + ··· + d m + b 1 + ··· + b l −→ R d 1 + ··· + d m + b 1 + ··· + b l , x (cid:55)→ σ ( x ) , we deﬁne its product space dual as

Original: [Equation]

Original: F. P RODUCT L OGITS

Original: One could be tempted to extend the logits through a combination of the ideas from this section and the previous one.

Original: More precisely, one could try to deﬁne a hyperplane as

Original: [Equation]

Original: Then one could again leverage the component-wise nature of ⊕ X and the linearity of (cid:104)(cid:105) in order to arrive at an expression decomposing over the components:

Original: [Equation]

Original: This expression is unfortunately not easy to handle as one cannot write this as the product of the smaller hyperplanes in each component.

Original: Especially obtaining theminimal distanceof a point x ∈ X to this hyperplane is not a simple task. Fortunately, it turns out that there is an easier approach: Deﬁnition F.3. The logits in the product of Riemannian manifold of constant sectional curvature are deﬁned as
Revised: However, we leave for future work the rigorous proof that the distance to geodesic hyperplanes in the positive curvature setting is given by this formula.

Original: [Equation]

Original: y

Original: [Equation]

Original: where a ∈ T x X ∼ =

Original: T x 1 B d 1 c − 1 × · · · × T x m B d m c − m × T ˜ x

Original: 1 R b 1 × · · · × T ˜ x

Original: m R b m and p ∈ X .

Original: Notice that in the limit where all curvatures c − i and ˜ c

Original: + i go to zero, we recover the expression

Original: [Equation]

Original: which are exactly the logits in the Euclidean space R d 1 + ··· + d m + b 1 + ··· + b l scaled by the factor 4 .

Original: We hence again recover the Euclidean counterpart when using Deﬁnition F.3 for the logits.

Original: F. L EFT M ATRIX M ULTIPLICATION AND M IDPOINTS
Revised: E M ORE E XPERIMENTAL D ETAILS

Original: Again, the missing ingredient for the deﬁnition of a left matrix multiplication is the notion of taking an average.

Original: As we will see in the following, the gyromidpoint can be easily extended component-wise while still preserving the desirable properties of its one-component counterpart.

Original: Deﬁnition F.4. For { x 1 , . . . , x n } ⊂ X and weights { α 1 , . . . , α n } ⊂ R the weighted gyromidpoint in the product manifold model is given by
Revised: We here present training details for the node classiﬁcation experiments.

Original: [Equation] We recover very similar properties to the Poincaré and spherical gyromidpoint: Lemma 9. For { x 1 , . . .
Revised: We closely follow the training and evaluation scheme from previous work, e.g. (Klicpera et al., 2019).

Original: , x n } ⊂ X , z ∈ X and weights { α 1 , . . .

Original: , α n } ⊂ R it holds that:

Original: • m X

Original: ( z ⊕ X x 1 , . . .

Original: , z ⊕ X x n ; α 1 , . .

Original: . , α n ) =

Original: z ⊕

Original: X m X ( x 1 , . . , x n ; α 1 , . . . , α n ) • d X ( x 1 , m c X ( x 1 , x 2 ; 12 , 12 )) = 12 d c X ( x 1 , x 2 ) • m X
Revised: We split the data into training, early stopping, validation and test set.

Original: ( x 1 , . . .

Original: , x n ; α 1 , . . .

Original: α n ) c − 1 ,..., c − m , c

Original: +1 ,..., c + l −→ 0 −−−−−−−−−−−−−−−→ m E ( x 1 , . . .

Original: , x n ; α 1 , . . .

Original: α n )

Original: Being already equipped with a scalar multiplication, we are thus ready to introduce the left matrix multiplication:

Original: Deﬁnition F.5.

Original: Given X ∈ R n × ( d 1 + ··· + d m + b 1 + ··· + b l ) with rows X i • ∈ X and A ∈ R n × n , the product left matrix multiplication Z =

Original: A ⊗ X X is deﬁned row-wise via

Original: [Equation]

Original: [Equation] Moreover, Z can be written in terms of the left matrix multiplications ofits components: [Equation] Notice how in contrast to the right matrix multiplication, the left matrix multiplication decomposes over the factors independently.
Revised: Namely we ﬁrst split the dataset into a known subset of size n known and an unknown subset consisting of the rest of the nodes.

Original: F. P RODUCT GCN

Original: Having gathered all needed tools in the previous section, we can introduce the architecture of the product GCN.

Original: Assume we are given a graph with node level features G

Original: =

Original: ( V, A, X ) where X ∈ R n × d and adjacency A ∈ R n × n .

Original: In order to map the Euclidean features X to the product manifold we apply the projection X (cid:55)→

Original: X 2 √ c

Original: || X

Original: || max component-wise for the hyperbolic components and the identity for the spherical components.

Original: Notice that we hence need to split the feature dimension into the separate components by writing d = (cid:80) ki =1 d i + (cid:80) li

Original: =

Original: b

Original: i , therefore deciding in advance what features belong to what component.

Original: We thus obtain a preprocessed ﬁrst embedding ˜ X = proj X ( X ) ∈ R n × ( d 1 + ··· + d m + b 1 + ··· + b l ) where each row ˜

Original: X i • ∈ X lives now in the product manifold.

Original: Forl ∈ { 0 , . . . , L − 2 } , the ( l + 1) -th layer of the product GCN is given by [Equation] where H (0) =
Revised: For

Original: ˜ X , σ is some non-linearity and ˆ A =

Original: ˜ D − 12 ˜

Original: A ˜ D − 12 .

Original: The ﬁnal layer is a product logit layer:

Original: [Equation] Notice how this architecture generalizes H -GCN and S -GCN since we obtain those by only using one hyperbolic or spherical component respectively. G E XPERIMENTAL DETAILS We here present training details for the node classiﬁcation experiments.
Revised: [Table]

Original: We closely follow the training and evaluation scheme from previous work, e.g. Klicpera et al. We split the data into training, early stopping, validation and test set. Namely we ﬁrst split the dataset into a known subset of size n known and an unknown subset consisting of the [Table]
Revised: all the graphs we use n known = 1500 except for MS Academics, where we use n known = 5000 .

Original: The known subset is further split into a training set consisting of 20 data points per label, an early stopping set of size 500 and a validation set of the remaining nodes.
Revised: The known subset is further split into a training set consisting of 20 data points per label, an early stopping set of size 500 and a validation set of the remaining nodes.

Original: Notice that the whole structure of the graph and all the node features are used in an unsupervised fashion since the embedding of a training node might for instance depend on the embedding of a node from the validation set.
Revised: Notice that the whole structure of the graph and all the node features are used in an unsupervised fashion since the embedding of a training node might for instance depend on the embedding of a node from the validation set.

Original: But when calculating the loss, we only provide supervision with the training data.
Revised: But when calculating the loss, we only provide supervision with the training data.

Original: The unknown subset serves as the test data and is only used for the ﬁnal evaluation of the model.
Revised: The unknown subset serves as the test data and is only used for the ﬁnal evaluation of the model.

Original: Hyperparameter-tuning is performed on the validation set.
Revised: Hyperparameter-tuning is performed on the validation set.

Original: We further use early stopping in all the experiments.
Revised: We further use early stopping in all the experiments.

Original: We stop training as soon as the early stopping cross entropy loss has not decreased in the last n patience = 200 epochs or as soon as we have reached n max = 2000 epochs.
Revised: We stop training as soon as the early stopping cross entropy loss has not decreased in the last n patience = 200 epochs or as soon as we have reached n max = 2000 epochs.

Original: The model chosen is the one with the highest accuracy score on the early stopping set.
Revised: The model chosen is the one with the highest accuracy score on the early stopping set.

Original: For the ﬁnal evaluation we test the model on 10 different data splits and report mean accuracy and bootstrapped conﬁdence intervals.We use the described setup for both the Euclidean and non-Euclidean models to ensure a fair comparison.
Revised: For the ﬁnal evaluation we test the model on 10 different data splits and report mean accuracy and bootstrapped conﬁdence intervals.

Original: Training details for the Euclidean Baselines We trained the Euclidean models with the hyperparameters chosen as reported in Klicpera et al.

Original: Namely, for GCN we use one hidden layer of size 64 , dropout on the embeddings and the adjacency of rate 0 .

Original: 5 as well as L 2 -regularization for the weights of the ﬁrst layer with λ = 0 .

Original: Only for Cora-ML we had to adjust the regularization factor λ to 0 .

Original: 002 to ensure similar scores as achieved in Klicpera et al.

Original: Summary of training details.

Original: All Non-Euclidean models usebiased-L2 regularization with α = 10 and λ = 2 e − 2 . Euclidean models used L2 regularization with the same parameter λ . We used a combination of dropout and dropconnect for  the  non-Euclideanmodels. All models have the same number of parameters.
Revised: We use the described setup for both the Euclidean and non-Euclidean models to ensure a fair comparison.

Original: We use 2 GCN layers, hidden dimension 64.

Original: Product models split hidden dimension into [32, 32] and also input features equally.

Original: Non-Euclidean models do not use additional non-linearities.

Original: Euclidean parameters use a learning rate of 0.01 for all models using ADAM.

Original: The curvatures are learned using gradient descent with lr 0.01 as well.

Original: We use early stopping: we ﬁrst train for a maximum of 2000 epochs, then we check every 200 epochs for improvement in the validation cross entropy loss; if that is not observed, we stop.

Original: Learned curvatures .
Revised: Learned curvatures .

Original: Citeseer:
Revised: Citeseer:

Original: HypGCN:
Revised: HypGCN:

Original: Trained curvature, average curvature over all runs: - [Table]
Revised: Trained curvature, average curvature over all runs: -1.057 +-0.

Original: Sphr GCN: Trained curvature, average curvature over all runs: [Table] [Table] : Trained curvatures, average curvatures: [
Revised: Sphr GCN: Trained curvature, average curvature over all runs: 0.951 +-0.

Original: [Table]

Original: [Table] ] CoraHypGCN: Trained curvature, average curvature: - [Table] Sphr GCN: Trained curvature, average curvature:
Revised: Prod

Original: [Table][Table]
Revised: [Table]

Original: Prod GCN: Trained curvature, average curvature: [-

Original: [Table]

Original: [Table]

Original: ] Pubmed

Original: Hyp GCN: Trained curvature, average curvature: 1.

Original: [Table][Table]
Revised: [Table]

Original: Sphr GCN: Trained curvature, average curvature: 0.896 +- 0. Prod GCN: Not training curvature, ﬁxed to [-1, -1] [Table] Hyp GCN: Trained curvature, average curvature: 1.26 +-
Revised: F G RAPH C URVATURE E STIMATION A LGORITHM

Original: [Table] Sphr GCN: Trained curvature, average curvature: 0.8 +- 0. Prod GCN: Not training curvature, ﬁxed to [-1, -1]
Revised: We used the following procedure to estimate the curvature of a dataset developed by Gu et al.

Original: Fix a node m ∈ G and sample two neighbouring nodes a, b ∈ G uniformly.
Revised: Fix a node m ∈ G and sample two neighbouring nodes a, b ∈ G uniformly.

Original: Further sample an additional reference node c ∈ G uniformly (again avoiding m = c ).
Revised: Further sample an additional reference node c ∈ G uniformly (again avoiding m = c ).

Original: Calculate ψ ( m ; a, b ; c ) =
Revised: Calculate ψ ( m ; a, b ; c ) =

Original: 1 2 d G ( a,b ) (cid:16) d 2 G ( a, m ) + d 2 G ( b,c )− (cid:16)
Revised: [Equation] 2 d G ( a,b ) [Equation]

Original: d 2 G (a,b )
Revised: d 2 G (

Original: +
Revised: [Equation] G [Equation] b

Original: [Table]
Revised: [Equation]

Original: Reiterate the above sampling n iter times and obtain an average curvature at node m .
Revised: Reiterate the above sampling n iter times and obtain an average curvature at node m .

Original: Do this procedure for every node m ∈ G .
Revised: Do this procedure for every node m ∈ G .