H YPERBOLIC N EURAL N ETWORKS ++
Anonymous authors
Paper under double-blind review
A BSTRACT
Hyperbolic spaces, which have the capacity to embed tree structures without distortion owing to their exponential volume growth, have recently been applied to machine learning to better capture the hierarchical nature of data.
In this study, we generalize the fundamental components of neural networks in a single hyperbolic geometry model, namely, the Poincaré ball model.
This novel methodology constructs a multinomial logistic regression, fully-connected layers, convolutional layers, and attention mechanisms under a uniﬁed mathematical interpretation, without increasing the parameters.
Experiments show the superior parameter efﬁciency of our methods compared to conventional hyperbolic components, and stability and outperformance over their Euclidean counterparts.
I NTRODUCTION
Shifting the arithmetic stage of a neural network to a non-Euclidean geometry such as a hyperbolic space is a promising way to ﬁnd more suitable geometric structures for representing or processing data.
Owing to its exponential growth in volume with respect to its radius (Krioukov et al., 2009; 2010), a hyperbolic space has the capacity to continuously embed tree structures with arbitrarily low distortion (Krioukov et al., 2010; Sala et al., 2018).
It has been directly utilized, for instance, to visualize large taxonomic graphs (Lamping et al., 1995), to embed scale-free graphs (Blasius et al., 2018), or to learn hierarchical lexical entailments (Nickel & Kiela, 2017).
Compared to the Euclidean space, a hyperbolic space shows a higher embedding accuracy under fewer dimensions in such cases.
Because a wide variety of real-world data encompasses some type of latent hierarchical structures (Katayama & Maina, 2015; Newman, 2005; Lin & Tegmark, 2017; Krioukov et al., 2010), it has been empirically proven that a hyperbolic space is able to capture such intrinsic features through representation learning (Krioukov et al., 2010; Ganea et al., 2018b; Nickel & Kiela, 2018; Tifrea et al., 2019; Law et al., 2019; Balazevic et al., 2019;
Gu et al., 2019).
Motivated by such expressive characteristics, various machine learning methods, including support vector machines (Cho et al., 2019) and neural networks (Ganea et al., 2018a; Gulcehre et al., 2018; Micic & Chu, 2018; Chami et al., 2019) have derived the analogous beneﬁts from the introduction of a hyperbolic space, aiming to improve the performance on advanced tasks beyond just representing data.
One of the pioneers in this area is Hyperbolic Neural Networks (HNNs), which introduced an easy-to-interpret and highly analytical coordinate system of hyperbolic spaces, namely, the Poincaré ball model, with a corresponding gyrovector space to smoothly connect the fundamental functions common to neural networks into valid functions in a hyperbolic geometry (Ganea et al., 2018a).
Built upon the solid foundation of HNNs, the essential components for neural networks covering the multinomial logistic regression (MLR), fully-connected (FC) layers, and Recurrent Neural Networks have been realized.
In addition to the formalism, the methods for graphs (Liu et al., 2019), sequential classiﬁcation (Micic & Chu, 2018), or Variational Autoencoders (Nagano et al., 2019; Mathieu et al., 2019; Ovinnikov, 2019; Skopek et al., 2020) are further constructed.
Such studies have applied the
Poincaré ball model as a natural and viable option in the area of deep learning.
Despite such progress, however, there still remain some unsolved problems and uncovered regions.
In terms of the network architectures, the current formulation of hyperbolic MLR (Ganea et al., 2018a) requires almost twice the number of parameters compared to its Euclidean counterpart.
This makes both the training and inference costly in cases in which numerous embedded entities should be classiﬁed or where large hidden dimensions are employed, such as in natural language processing.
The lack of convolutional layers must also be mentioned, because their application is now ubiquitous and is no longer limited to the ﬁeld of computer vision.
For the individual functions that are commonly used in machine learning, the split and concatenation of vectors have yet to be realized in a hyperbolic space in a manner that can fully exploit such space and allow sub-vectors to achieve a commutative property.
Additionally, although several types of closed-form centroids in a hyperbolic space have been proposed, their geometric relationships have not yet been analyzed enough.
Because a centroid operation has been utilized in many recent attention-based architectures, the theoretical background for which type of hyperbolic centroid should be used would be required in order to properly convert such operations into the hyperbolic geometry.
Based on the previous analysis, we reconsider the ﬂow of several extensions to bridge Euclidean operations into hyperbolic operations and construct alternative or novel methods on the Poincaré ball model.
Speciﬁcally, the main contributions of this paper are summarized as follows:
We reformulate a hyperbolic MLR to reduce the number of parameters to the same level as a Euclidean version while maintaining the same range of representational properties.
We further exploit the knowledge of 1 as a replacement of an afﬁne transformation and propose a novel generalization of the FC layers that can more properly make use of the hyperbolic nature compared with a previous research (Ganea et al., 2018a).
We generalize the split and concatenation of coordinates to the Poincaré ball model by setting the invariance of the expected value of the vector norm as a criterion.
By combining 2 and 3, we further deﬁne a novel generalization scheme of arbitrary dimensional convolutional layers in the Poincaré ball model.
We prove the equivalence of the hyperbolic centroids deﬁned in three different hyperbolic geometry models, and expand the condition of non-negative weights to entire real values.
Moreover, integrating this ﬁnding and previous contributions 1,
and 3, we give a theoretical insight into hyperbolic attention mechanisms realized in the Poincaré ball model.
We experimentally demonstrate the effectiveness of our methods over existing HNNs and Euclidean equivalents based on a performance test of MLR functions and experiments with Set Transformer (Lee et al., 2019) and convolutional sequence to sequence modeling (Gehring et al., 2017).
H YPERBOLIC G EOMETRY
Riemannian geometry.
An n -dimensional manifold M is an n -dimensional topological space that can be linearly approximated to an n -dimensional real space at any point x ∈ M , and each local linear space is called a tangent space T x M .
A Riemannian manifold is a pairing of a differentiable manifold and a metric tensor ﬁeld g as a function of each point x , which is expressed as ( M , g ) .
Here, g deﬁnes an inner product on each tangent space such that ∀ u , v ∈ T x M , (cid:104) u , v (cid:105) x
= u (cid:62) g x v , where g x is a positive deﬁnite symmetric matrix deﬁned on T x M .
The norm of a tangent vector derived from the inner product is deﬁned as (cid:107) v (cid:107) x = (cid:112) |(cid:104) v , v (cid:105) x | .
A metric tensor g x provides local information regarding the angle and length of the tangent vectors in T x M , which induces the global length of the curves on M through an integration.
The shortest path connecting two arbitrary points on M at a constant speed is called a geodesic, the length of which becomes the distance.
Along a geodesic where one of the endpoints is x , the function projecting a tangent vector v ∈ T x M as an initial velocity vector onto M is denoted as an exponential map exp x , and its inverse function is called a logarithmic map log x .
In addition, the concept of parallel transport P x → y : T x M → T y M is generalized to the specially conditioned unique linear isometry between two tangent spaces.
For more details, please refer to Spivak (1979); Petersen et al.
; Andrews & Hopper (2010).
Note that, in this study, we equate g with g x if g x is constant, and denote the Euclidean inner product, norm, and unit vector for any real vector u , v ∈ R n as (cid:104) u , v (cid:105) , (cid:107) v (cid:107) , and [ v ] = v / (cid:107) v (cid:107) , respectively.
Hyperbolic space.
A hyperbolic space is a Riemannian manifold with a constant negative curvature, the coordinates of which can be represented in several isometric models.
The most basic model is an n -dimensional hyperboloid model, which is a hypersurface H nc in an ( n + 1) -dimensional
Minkowski space R n + composed of one time-like axis and n space-like axes.
The manifolds of Poincaré ball model B nc and Beltrami-Klein model K nc are the projections of the hyperboloid model onto the different n -dimensional space-like hyperplanes, as depicted in Figure 1.
For their mathematical deﬁnitions and the isometric isomorphism between their coordinates, see Appendix A.
Poincaré ball model.
The n -dimensional Poincaré ball model of a constant negative curvature − c is deﬁned by ( B nc , g c ) , where B nc = { x ∈ R n | c (cid:107) x (cid:107) 2 < 1 } and g c x =
( λ c x ) 2 I n .
Here, B nc is an open ball of radius c − 12 , and λ c x = 2(1 − c (cid:107) x (cid:107) 2 ) − 1 is a conformal factor, which induces the inner product (cid:104) u , v (cid:105)
c x
= ( λ c x ) 2 (cid:104) u , v (cid:105) and
(cid:107) v (cid:107) c x = λ c x (cid:107) v (cid:107) for u , v ∈ T x B nc .
The exponential, logarithmic maps and parallel transport are denoted as exp c x , log c x and P c x → y , respectively, as shown in Appendix C.
To operate the coordinates as vector-like mathematical objects, the
Möbius gyrovector space provides an algebra that treats them as gyrovectors, equipped with various operations including the generalized vector addition, that is, a noncommutative and non-associative binary operation called the Möbius addition ⊕ c (Ungar, 2009).
lim c → 0 ⊕ c converges to + in connection with a Euclidean geometry, the curvature of which is zero.
For more details, see Appendix B.
Poincaré hyperplane.
As a speciﬁc generalization of a hyperplane into Riemannian geometry, Ganea et al.
(2018a) derived a Poincaré hyperplane
˜ H c a , p , which is the set of all geodesics containing an arbitrary point p ∈ B nc and orthogonal to an arbitrary tangent vector a ∈ T p B nc , based on the Möbius gyrovector space.
As shown in Appendix C.2, they also extended the distance d c between two points in B nc into the distance from a point in B nc to a Poincaré hyperplane in a closed form expression.
H YPERBOLIC N EURAL N ETWORKS ++
[Figure]
Considering one bias point q a ,r per one discriminative hyperplane solves this over-parameterization.
Aiming to overcome the difﬁculties discussed in Section 1, we build a novel scheme of hyperbolic neural networks in the Poincaré ball model.
The core concept is re-generalization of (cid:104) a , x (cid:105)− b type equations with no increase in the number of parameters, which has the potential to replace any afﬁne transformation based on the same mathematical principle.
Speciﬁcally, this section starts from the reformulation of the hyperbolic MLR, from which the variants to the
FC, convolutional, and multi-head attention layers are derived.
Several other modiﬁcations are also proposed to support neural networks with broad architectures.
3. U NIDIRECTIONAL REPARAMETERIZATION OF HYPERBOLIC MLR LAYER
Given an input x ∈ R n , MLR is an operation used to predict the probabilities of all target outcomes k ∈ { 1 , 2 , ..., K } for the objective variable y as a log-linear model and is described as follows: p ( y = k | x ) ∝ exp ( v k ( x )) , where v k ( x ) = (cid:104) a k , x (cid:105) − b k , a k ∈ R n , b k ∈ R .
Circumvention of the double vectorization.
To generalize the linear function v k to the Poincaré ball model, Ganea et al.
(2018a)
ﬁrst re-parameterized the scalar term b k as a vector p k ∈ R n in the form (cid:104) a k , x (cid:105) − b k = (cid:104) a k , − p k + x (cid:105) , where b k = (cid:104) a k , p k (cid:105) , and then discussed the properties which must be satisﬁed when such vectors become Möbius gyrovectors.
However, this causes an undesirable increase in the parameters from n + 1 to 2 n in each class k .
As illustrated in Figure(a), this reformulation is redundant from the viewpoint that there exist countless choices of p k to determine the same discriminative hyperplane H a k ,b k = { x ∈ R n | (cid:104) a k , x (cid:105)
− b k = 0 } .
Because the key of this step is to replace all variables with vectors attributed to the same manifold, we introduce another scalar parameter r k ∈ R instead, which makes the bias vector q a k ,r k parallel to a k :
[Equation]
, where q a k
[Equation]
One possible realization of p k is adopted to reduce the previously mentioned redundancies without a loss of generality or representational properties compared to the original afﬁne transformation, and
[Figure]
induces another notation: ¯ H
[Equation] [Equation]
, −
[Equation]
= 0 }
= H a k ,
[Equation]
k (cid:107) a k (cid:107) .
Based on distance d from a point to a hyperplane, Equation 2 can be rewritten as with Lebanon & Lafferty (2004) in the following form: (cid:104)
[Equation] [Equation]
(cid:107)
[Equation]
k (cid:107) , which decomposes the inner into the product of the norm of an orientation vector a k and the signed distance between an input vector x ∈ R n and the hyperplane ¯ H a
,r k .
Unidirectional Poincaré MLR.
Based on the observation that q a k ,r k starts from the origin and the concept of Poincaré hyperplanes, we can now generalize v k for x , q a k ,r k ∈ B nc and a k ∈ T q a k,rk B nc : v k (
[Equation]
=
[Equation] [Equation]
where q a k ,r k = exp c 0 ( r k [ a k ])
[Equation]
which are shown in Figure 2 (b).
Importantly, the circular reference between a k ∈ T q a k,rk B nc and q a k ,r k can be unraveled by considering the tangent vector at the origin, z k ∈ T 0 B nc , from which a k is parallel transported by P c x
→ y : T x B nc
→ T y B nc described in Appendix C.3 as follows:
[Equation]
Combining Equations 3, 5, and 23, we conclude the derivation of the unidirectional re-generalization of MLR, the parameters of which are r k ∈ R and z k ∈ T 0 B nc = R n for each class k :
[Equation]
For more detailed deformation, see Appendix D.1.
Note that we recover the form of the standard
Euclidean MLR in lim c
→ 0 v k ( x ) = 4( (cid:104) a k , x (cid:105) − b k ) , which is proven in Appendix D.2.
R EFORMULATING FC LAYERS TO PROPERLY EXPLOIT THE HYPERBOLIC PROPERTIES
We next discuss the FC layers, described as a simple afﬁne transformation y =
Ax − b , in an element-wise manner with respect to the output space as y k = (cid:104) a k , x (cid:105) − b k , where x , a k ∈ R n and b k ∈ R .
This can be interpreted as an operation that linearly transforms the input x and treats the output score y k as the coordinate value at, or the signed distance from the hyperplane containing the origin and orthogonal to, the k -th axis of the output space R m .
Therefore, combining them with a generalized linear transformation, as described in Section 3.1, we can now generalize the FC layers:
Poincaré FC layer.
Given an input x ∈ B nc , with the generalized linear transformation v k in Equation 6 and the parameters composed of Z = { z k ∈ T 0 B nc = R n } mk =1 , which is a generalization of A and r = { r k ∈ R } mk =1 representing the bias terms, the Poincaré FC layer outputs the following:
[Equation]
1 , where w :=
( c − 12 sinh (cid:0) √ c v k ( x ) (cid:1) )
mk =1 .
It can be proven that the signed distance from y to each Poincaré hyperplane containing the origin, and orthogonal to the k -th axis, equals v k ( x ) , as shown in Appendix D.3, satisfying the aforementioned properties.
We also recover a FC layer in lim c → 0 y k
= 4 ( (cid:104) a k , x (cid:105) − r k (cid:107) a k (cid:107) ) .
Comparison with a previous method.
Ganea et al.
(2018a) proposed a hyperbolic FC layer operating a matrix-vector multiplication in a tangent space and adding a bias through the following
Möbius addition: y = exp c
( A log c 0 ( x )) ⊕ c b , which indicates that the discriminative hyperplane determined in T 0 B mc is projected back to B mc by the exponential map at the origin.
However, such a surface is no longer a Poincaré hyperplane, except for b = 0 .
Moreover, the basic shape of the contour surfaces in the output space B mc is determined only by the orientation of each row vector a k in A , whereas their norms and a bias term b contribute to the total scale and shift.
Conversely, the parameters in our method cooperate to realize more various contour surfaces.
Notably, discriminative hyperplanes become Poincaré hyperplanes, i .
e ., the set of all geodesics orthogonal to the orientation z k and containing a point exp c 0 ( r k [ z k ]) .
As shown in Figure 3, the input space B nc is separated in a more meaningful manner as a hyperbolic space for each dimension of the output space B mc .
3. R EGULARIZING SPLIT AND CONCATENATION
Split and concatenation are essential operations for realizing small process branches in parallel or combining feature vectors.
However, in the Poincaré ball model, merely splitting the coordinates lowers the norms of the output gyrovectors and limits the representational power, and concatenating them is invalid because the norm of the output can easily exceed the domain of the ball.
One simple solution is to conduct an operation in the tangent space.
The aforementioned problem regarding a split operation, however, remains.
Moreover, as the number of inputs to be concatenated increases, the output gyrovector approaches the boundary of the ball even if the norm of each input is adequately small.
The norm of the gyrovector is crucial in the Poincaré ball model owing to its metric.
Therefore, reﬂecting the orientation of inputs while preserving the scale of the norm is considered to be desirable.
Generalization criterion.
In Euclidean neural networks, keeping the variance of feature vectors constant is an essential criterion (He et al., 2015).
As an analogy, keeping the expected values of the norms constant is a worthy criterion in the Poincaré ball because the norm of any Möbius gyrovector is upper-bounded by the ball radius and the variance of the coordinates cannot necessarily remain intact when the dimensions in each layer vary.
Such a replacement of the statistic invariance target from each coordinate to the norm is also suggested by Becigneul & Ganea (2019).
To satisfy this criterion, we propose the following generalization scheme with a scalar coefﬁcient β n = B( n 2 , 12 ) .
Poincaré β -split.
First, the input x ∈ B nc is split in the tangent space with integers s.t.
(cid:80) Ni =
1 n i = n : x (cid:55)→ v = log c 0 ( x ) =
( v (cid:62)
[Equation]
R n 1 , . .
. , v (cid:62) N ∈ R n N ) (cid:62) .
Each split tangent vector is then properly scaled and projected back to the Poincaré ball as follows:
v i (cid:55)→ y i = exp c 0 (cid:0) β n i β − 1 n v i (cid:1) .
Poincaré β -concatenation.
Likewise, the inputs { x i ∈ B n i c } Ni =1 are ﬁrst properly scaled and concatenated in the tangent space, and then projected back to the Poincaré ball in the following manner: x i (cid:55)→
[Equation]
We prove the previously mentioned properties under a certain assumption in Appendix D.4.
One can also conﬁrm that the Poincaré β -concatenation is the inverse function of the Poincaré β -split.
Discussion about the concatenation.
Ganea et al.
(2018a) generalized a vector concatenation under the premise that the output must be followed by an FC layer, but such an assumption possibly limits its usage.
Furthermore, it requires Möbius additions N − 1 times sequentially due to the noncommutative and non-associative properties of the Möbius addition, which incurs a heavy computational cost and an unbalanced priority in each input gyrovector.
Alternatively, our method with a pair of exponential and logarithmic maps has a lower computational cost regardless of N and treats every input fairly.
3. A RBITRARY DIMENSIONAL CONVOLUTIONAL LAYER
The activation of D -dimensional convolutional layers with kernel sizes of { K i } Di =1 is generally described as an afﬁne transformation y k
= (cid:104) a k , x (cid:105) − b k for each channel k , where x ∈ R nK is an input vector per pixel, and is a concatenation of K = (cid:81)
i K i feature vectors contained in a receptive ﬁeld of the kernel.
This notation also includes a dilated operation.
It is now natural to generalize the convolutional layers with Poincaré β -concatenation and a Poincaré FC layer.
Poincaré convolutional layer.
At each pixel in the given feature map, the gyrovectors { x s ∈ B nc } Ks = contained in a receptive ﬁeld of the kernel are concatenated into a single gyrovector x ∈ B nKc in the manner proposed in Section 3.3, which is then operated in the same way as a Poincaré FC layer.
3. A NALYSIS OF HYPERBOLIC ATTENTION MECHANISMS IN THE P OINCARÉ BALL MODEL
As preparation for constructing hyperbolic attention mechanisms, it is necessary to theoretically consider the midpoint operation of multiple coordinates in a hyperbolic space.
For the Poincaré ball model and Beltrami-Klein model, Ungar (2009) proposed the Möbius gyromidpoint and Einstein gyromidpoint built upon the framework of gyrovector spaces, respectively, which are represented in different coordinate systems but are geometrically the same, as shown in Appendix D.5.
On the other hand, Law et al.
(2019) proposed another type of hyperbolic centroid based on the minimization problem of the squared Lorentzian distance deﬁned in the hyperboloid model.
Based on the above situation, for the major concern of which formulation to utilize, we proved the following theorem.
Theorem 1.
(The equivalence of three hyperbolic midpoints)
The Möbius gyromidpoint, Einstein gyromidpoint, and the centroid of the squared Lorentzian distance exactly match each other, which indicates they are the same midpoint operations projected on each manifold.
The proofs are given in Appendix D.5 and D.6.
With this theorem, we can now exploit the Möbius gyromidpoint as a uniﬁed option to realize hyperbolic attention mechanisms.
Moreover, we further generalized the Möbius gyromidpoint by extending the condition of non-negative weights to entire real values by regarding a negative weight as an additive inverse operation:
The midpoint ¯ b ∈ B nc of
Möbius gyrovectors { b i ∈ B nc } Ni =1 with the real scalar weights { ν i ∈ R } Ni =1 is given by
[Equation]
which is shown in Appendix D.8.
Note that the sum of weights does not need to be normalized to one because any scalar scale is cancelled between the numerator and denominator.
On the basis of this insight, in the following, we describe the construction of a multi-head attention as a speciﬁc example, aiming at a general approach that can be applied to other arbitrary attention schemes.
Multi-head attention.
Given a source sequence S ∈ R L s × n of length L s and target sequence
T ∈ R L t × m of length L t , the module ﬁrst projects the target onto query Q ∈ R L t × hd and the source onto key K ∈ R L s × hd and value V ∈ R L s × hd with the corresponding FC layers.
These are split into d -dimensional vectors of h heads, which is followed by a similarity function between Q i and K i producing a weight Π i = { softmax( d − 12 q it (cid:62) K i ) }
L t t =1 for 1 ≤ i ≤ h .
The weights are utilized to aggregate V i into a centroid, giving X i
=
Π i V i .
Finally, the features in all heads are concatenated.
Poincaré multi-head attention.
Given the source and target as sequences of gyrovectors, they are projected with three Poincaré FC layers, followed by Poincaré β -splits to produce Q i = { q it ∈
B dc } L t t =1 , K i = { k is ∈ B dc } L s s =1 and V i
= { v is ∈ B dc } L s s =1 .
Applying a similarity function f c and activation g , each weight π
it,s = g
( f c ( q it , k is )) is obtained and the values are aggregated as follows:
x it = 1 ≤ s ≤ L s (cid:2) v is , π it,s (cid:3) c .
Finally, the features in all heads are Poincaré β -concatenated.
For the similarity function f c , there are mainly two choices to exploit.
One is the inner product in the tangent space indicated by Micic & Chu (2018), which is the naive generalization of the Euclidean version.
Another choice is based on the distance of two points: f c ( q , k ) = − τd c ( q , k ) − γ , where τ is an inverse temperature and γ is a bias parameter, which was proposed by Gulcehre et al.
As for the activation g , g ( x ) = exp ( x ) is the most basic option because it turns to be a softmax operation due to the property of gyromidpoint.
Gulcehre et al.
(2018) also suggested g ( x ) =
σ ( x ) .
In light of the property of the generalized gyromidpoint, g as an identity function is also exploitable.
E XPERIMENTS
In this section, we evaluate our methods in comparisons with HNNs and Euclidean counterparts.
The implementation of hyperbolic architectures is based on the Geoopt (Kochurov et al., 2020).
4. V ERIFICATION OF THE MLR CLASSIFICATION CAPACITY
We ﬁrst evaluated the performance of our unidirectional Poincaré MLR on the same conditioned experiment designed for the MLR of HNNs, that is, a sub-tree classiﬁcation on the Poincaré ball model.
In this task, the Poincaré embeddings of the WordNet noun hierarchy (Nickel & Kiela, 2017)
[Table]
are utilized as the data set, which contains 82,115 nodes and 743,241 hypernymy relations.
We pre-trained the Poincaré embeddings of the same dimensions as the experimental settings in HNNs, i .
e ., two, three, ﬁve, and ten dimensions, using the open-source implementation 1 to extract several sub-trees whose root nodes are certain abstract hypernymies, e .
g ., animal.
For each sub-tree, MLR layers learn the binary classiﬁcation to predict whether each given node is included.
All nodes are divided into 80% training nodes and 20% testing nodes.
We trained each model for 30 epochs using
Riemannian Adam (Becigneul & Ganea, 2019) with a learning rate of 0.001 and a batch size of 16.
The F1 scores for the test sets are shown in Table 1.
From the results, we can conﬁrm the tendency of the hyperbolic MLRs to outperform the Euclidean version in all settings, which illustrates that MLR considering the hyperbolic geometry are better suited to the hyperbolic embeddings.
In particular, our parameter-reduced approach obtains the same level of performance as a conventional hyperbolic
MLR in a more stable training, as can be seen from the relatively narrower conﬁdence intervals.
4. A MORTIZED CLUSTERING OF MIXTURE OF G AUSSIANS WITH S ET T RANSFORMERS
For the evaluation of a Poincaré multi-head attention, we utilize Set Transformer, which we consider is a proper test case to eliminate the implicit inﬂuence of unessential operations, e .
g ., positional encoding.
The task is an amortized clustering of a mixture of Gaussians (MoG).
In each sample in a mini-batch, models take hundreds of two-dimensional points randomly generated by the same
K -component MoG, and directly estimate all the parameters, i .
e ., the ground truth probabilities, means, and standard deviations, in a single forward step.
We basically follow the model architectures and experimental settings of the ofﬁcial implementation 2 , except that we employed the hyperbolic
Gaussian distribution (Ovinnikov, 2019) as well as the Euclidean distribution aiming to verify the performance of the hyperbolic architectures both for the ordinary settings and for their desirable data distributions.
When the hyperbolic models need to deal with Euclidean coordinates, the inputs or outputs are projected by an exponential map or logarithmic map, respectively, with a scalar parameter for scaling the values to the ﬁxed-radius Poincaré ball B n 1 .
Note that we omit ReLU activations for our models because the hyperbolic operations are inherently non-linear.
We also remove normalization layers because there does not exist enough research on the normalizing criterion in the hyperbolic space and existing methods possibly reduce the representational power of gyrovectors by neutralizing their norms.
For the hyperbolic attentions, based on a preliminary experiment, we choose to utilize the distance based similarity function and exponential activation.
The results are shown in Table 2.
For the Euclidean distribution, our models achieved almost the same performance as Set Transformers, while the training of those without Layer Normalization for
[Table]
the same conditioned comparison often failed under all settings.
This result suggests the intrinsic normalization properties of our methods, which we attribute to the computation using vector norms.
For the hyperbolic distribution, our models outperformed the Euclidean counterparts with an order of magnitude smaller conﬁdence intervals, which indicates that our hyperbolic architectures are indeed suited to their assumed data distribution.
4. C ONVOLUTIONAL SEQUENCE TO SEQUENCE MODELING
Finally, we experimented with the convolutional sequence-to-sequence modeling task for machine translation of WMT’17 English-German (Bojar et al., 2017).
Because the architecture is composed of convolutional layers and attention layers, the hyperbolic version of which has already been veriﬁed in Section 4.2, it would provide a comparison focusing on convolutional operations.
It also has a practical aspect as a task of natural language processing, in which lexical entities are known to form latent hierarchical structures.
We follow the open-source implementation of Fairseq (Ott et al., 2019), where preprocessed training data contains 3.96M sentence pairs with 40K sub-word tokenization in each language.
In our hyperbolic models, feature vectors are completely treated as
Möbius gyrovectors because token embeddings can be learned directly on the Poincaré ball model.
Note that the inputs for the sigmoid functions in Gated Linear Units are logarithmically mapped just like hyperbolic Gated Recurrent Units proposed by Ganea et al.
(2018a).
We train various scaled-down models to verify the representational capacity of our hyperbolic architectures, with
Riemannian Adam for 100K iterations.
For more implementation details, please check Appendix E.
[Table]
The results are shown in Table 3.
Our model demonstrates the signiﬁcant improvements compared to the usual Euclidean models in the fewer dimensions, which reﬂects the immense embedding capacity of hyperbolic spaces.
On the other hand, there is no salient differences observed in higher dimensions, which implies that the Euclidean models with higher dimensions than a certain level can obtain a sufﬁcient computational complexity through the optimization.
This would ﬁll the gap with the representational properties of hyperbolic spaces.
It also implies that the proper construction of neural networks with the product space of multiple small hyperbolic spaces using our methods has the potential for the further improvements even in higher dimensional architectures.
C ONCLUSION
We showed a novel generalization and construction scheme of the wide range of hyperbolic neural network architectures in the Poincaré ball model, including a parameter-reduced MLR, geodesicaware FC layers, convolutional layers and attention mechanisms.
These were achieved under a uniﬁed mathematical backbone based on the concepts of Riemannian geometry and the Möbius gyrovector space, which endow our hyperbolic architectures with theoretical consistency.
Through the experiments, we veriﬁed the effectiveness of our approaches from diversiﬁed tasks and perspectives, such as an embedded sub-tree classiﬁcation, amortized clustering of distributed points both on the
Euclidean space and the Poincaré ball model, and neural machine translation.
We hope that this study will pave the way for future research in the ﬁeld of geometric deep learning
A H YPERBOLIC G EOMETRY
In this section, we review the deﬁnition of the hyperbolic geometry models other than the Poincaré ball model and the relationships between their coordinates.
Hyperboloid model.
The n -dimensional hyperboloid model is a hypersurface in an ( n + 1) - dimensional Minkowski space R n + , which is equipped with an inner product (cid:104) x , y (cid:105)
L = x (cid:62) g L y for ∀ x , y ∈ R n + , where g L = diag ( − 1 , 1 (cid:62) n ) .
Given a constant negative curvature − c , the manifold of the hyperboloid model is deﬁned by H nc = { x = ( x
[Equation]
, x n ) (cid:62)
[Equation]
R
[Equation]
[Equation] Note that, in this standard ( n + 1) -dimensional coordinate system, the metric tensor as a positive deﬁnite matrix for the n -dimensional hyperboloid manifold cannot be deﬁned.
Instead, when the hyperboloid model is represented in a speciﬁc n -dimensional coordinates, e .
g ., hyperbolic polar coordinates, then its metric tensor has the corresponding representation as the n -dimensional positive deﬁnite matrix.
Isometric isomorphism with the Poincaré ball model.
The bijection between an arbitrary point h
= ( z, k (cid:62) )
(cid:62) ∈ H nc and its unique corresponding point b ∈ B nc , depicted in Figure 1, is given by the following:
[Equation]
Beltrami-Klein model.
The n -dimensional Beltrami-Klein model of a constant negative curvature − c is deﬁned by ( K nc , ˆ g c ) , where K nc = { x ∈ R n | c (cid:107) x (cid:107) 2 < 1 } and ˆ g c x = (1 − c (cid:107) x (cid:107) 2 ) − 1 I n + (1 − c (cid:107) x (cid:107) 2 ) − 2 xx (cid:62) .
Here, K nc is an open ball of radius 1 / √ c .
Isometric isomorphism with the Poincaré ball model.
The bijection between an arbitrary point n ∈ K nc and its unique corresponding point b ∈ B nc , depicted in Figure 1, is given by the following:
[Equation]
B M ÖBIUS GYROVECTOR SPACE
In this section, we brieﬂy introduce the concept of the Möbius gyrovector space, which is a speciﬁc type of gyrovector spaces.
For a rigorous theoretical and detailed mathematical background of this system, please refer to Ungar (2005; 2009; 2001; 2012).
A gyrovector space is an algebraic structure that endows the points in a hyperbolic space with vector-like properties based on a special concept called a gyrogroup.
This gyrogroup is similar to ordinary vector spaces that provides a Euclidean space with the well-known vector operations based on the notion of groups.
As a particular example in physics, this helps to understand the mathematical structure of the Einstein’s theory of special relativity where no possible velocity vectors including the sum of velocities in an arbitrary additive order can exceed the speed of light (Ungar, 2008; 2013).
Because hyperbolic geometry has several isometric models, a gyrovector space also has some variants where the Möbius gyrovector space is a variant for the Poincaré ball model.
As an abstract mathematical system, a gyrovector space is constructed through the following steps: (1) Start from a set G .
(2) With a certain binary operation ⊕ , create a tuple called a groupoid, or magma ( G, ⊕ ) .
(3) Based on ﬁve axioms, deﬁne a speciﬁc type of magma as a gyrogroup.
These axioms include several important properties of gyrovector spaces, such as the left gyroassociative law and an operator called a gyrator gyr : G × G → Aut( G, ⊕ ) , which generates an automorphism
Aut( G, ⊕ ) (cid:51) gyr
[ x , y ] : G → G given by z (cid:55)→ gyr
[ x , y ] z , called a gyration, from two arbitrary points x and y ∈ G .
The notion of the gyrocommutative law and gyrogroup cooperation are given in this step.
(4) Adding ten more axioms related to the statements about a real inner product space and a scalar multiplication ⊗ , the gyrovector space ( G, ⊕ , ⊗ ) is thus deﬁned.
Some of the important properties of a gyrovector space are listed below.
Here, x , y , z ∈ G .
Gyroassociative laws.
Although the binary operation ⊕ is not necessarily associative in general, it obeys the left gyroassociative law x ⊕ ( y ⊕ z ) =
( x ⊕ y )
⊕ gyr[ x , y ]
z and right gyroassociative law ( x ⊕ y )
⊕ z
= x ⊕ ( y ⊕ gyr
[ y , x ] z ) .
These equations also provide a general closed-form expression of the gyrations: gyr[ x , y ]
z = (cid:9) ( x ⊕ y ) ⊕ ( x ⊕ ( y ⊕ z )) .
Cases in which gyrations become identity maps.
If at least one element for gyr is 0 ∈ G , the gyrations become an identity map I : gyr
[ x , 0 ]
= gyr[ 0 , x ]
= I .
With the loop properties of the gyrations given by gyr[ x , y ] =
gyr[ x ⊕ y , y ]
= gyr
[ x , y ⊕ x ] , many other cases can be also derived.
Gyrocommutative law.
Although a binary operation ⊕ is not necessarily commutative in general, if it obeys the equation x ⊕ y = gyr
[ x , y ]( y ⊕ x ) , the gyrogroup is called gyrocommutative.
Gyrogroup cooperation.
Regarding ⊕ as the primal binary addition, the second binary addition in
G is deﬁned as the gyrogroup cooperation (cid:1) , which is given by x (cid:1) y
= x ⊕ gyr[ x , (cid:9) y ] y .
This has duality symmetries with the ﬁrst binary operation ⊕ , such that x ⊕ y =
x (cid:1) gyr
[ x , y ] y .
In addition, corresponding to the left cancellation law (cid:9) x ⊕ ( x ⊕ y ) =
y inherent in ⊕ , the gyrogroup cooperation induces two types of the right cancellation laws: ( x ⊕ y ) (cid:12) y
=
( x (cid:1) y ) (cid:9)
y
= x .
In this formalism, the Möbius gyrovector space is then deﬁned as ( B nc , ⊕ c , ⊗ c ) , where B nc is as previously introduced in Section 2, and ⊕ c and ⊗ c are as shown in the following subsections.
B. M ÖBIUS ADDITION
In the Möbius gyrovector space
[Equation]
the primary binary operation is denoted as the Möbius addition ⊕ c : B nc × B nc → B nc , which is a noncommutaive and nonassociative addition, given by the following:
[Equation]
B. M ÖBIUS G YRATOR
The expression of gyrations in the Möbius gyrovector space can be expanded using the equation of the Möbius addition ⊕ c , which is described by Ungar (2009) as follows:
[Equation]
By writing down all the special operators ⊕ c for the gyrovectors in B nc into the normal vector operations, the expression of the gyrations can be now seen as a general function for the any real vector z ∈ R n .
Indeed, gyrations are extended to invertible linear maps of R n (Ungar, 2009).
The Möbius gyrator endows the Möbius gyrovector space with a gyrocommutative nature.
B. M ÖBIUS COADDITION
The gyrogroup cooperation in the Möbius gyrovector space is called the Möbius coaddition, and is given by the following:
[Equation]
With the gamma factor γ x = ( (cid:112) 1 − c (cid:107) x (cid:107) 2 ) − 1 for x ∈ B n c , this is also described in the following manner:
[Equation]
Note that the Möbius coaddition is not associative but is commutative.
B. M ÖBIUS SCALAR MULTIPLICATION
The Möbius scalar multiplication for x ∈ B nc and r ∈ R is given by the following:
[Equation]
In terms of the Riemannian geometry, the Möbius scalar multiplication adjusts the distance of x from the origin by the scalar multiplier r .
The expressions of the logarithmic map log c x and distance in the
Möbius gyrovector space are described in the following subsections.
C P OINCARÉ BALL MODEL
Owing to the algebraic structure provided by the Möbius gyrovector space, many properties related to the geometry of the Poincaré ball model can be described in implementation-friendly closed-form expressions.
C. E XPONENTIAL AND LOGARITHMIC MAPS
The exponential map exp c x : T x B nc
→ B nc is described in (Ganea et al., 2018a, Lemma 2) as follows:
[Equation]
The logarithmic map log c x = (exp c x ) − 1 : B nc → T x B nc is also given by the following:
[Equation]
C.
[Equation]
D ISTANCE C.2. P OINCARÉ DISTANCE BETWEEN TWO ARBITRARY POINTS
The distance function d c is originally and preliminary deﬁned as a binary operation for indicating the distance between two arbitrary points x , y ∈ B nc .
Based on the notion of the Möbius addition, the distance d c : B nc × B nc → R is succinctly described as follows:
[Equation]
Despite the noncommutative aspect of the Möbius addition ⊕ c , this distance function in Equation 19 becomes commutative thanks to the commutative aspect of the Euclidean norm of the Möbius addition, which is expressed as follows:
[Equation]
C.2.
D ISTANCE FROM A POINT TO P OINCARÉ HYPERPLANE
In the Euclidean geometry, the generalized concept of two-dimensional plane to a higher dimensional space R n is a hyperplane containing an arbitrary point p ∈ R n and is the set of all straight lines orthogonal to an arbitrary orientation vector a ∈ R n .
Because straight lines in Euclidean spaces are geodesics in terms of the Riemannian geometry, a hyperplane can be generalized as another
Riemannian manifold M n such that the hyperplane contains an arbitrary point p ∈ M n and is the set of all geodesics orthogonal to an arbitrary orientation vector at p , namely, the tangent vector a ∈ T p M n .
This concept in the Poincaré ball model has been rigorously deﬁned in (Ganea et al., 2018a, Deﬁnition 3.1) for p ∈ B nc , a ∈ T p B nc as follows:
[Equation]
Note that { a } ⊥ is the set of all tangent vectors at p and orthogonal to a .
Ganea et al.
(2018a) have proven the closed-form description of the distance from a point x ∈ B nc to an arbitrary Poincaré hyperplane
˜ H c a , p by considering the minimum distance between x and any point in ˜
H c
a , p :
[Equation]
C. P ARALLEL TRANSPORT
The concept of a parallel transport is traditionally derived from differential geometry.
In the hyperbolic geometry, the gyrovector space provides the algebra to formulate the parallel transport of a gyrovector (Ungar, 2012).
When a gyrovector (cid:9) c x ⊕ c w ∈ B nc rooted at a point x ∈ B nc is transported parallel to another gyrovector (cid:9)
c y ⊕ c z ∈ B nc rooted at a point y ∈ B nc along a geodesic connecting x and y , the equation below is satisﬁed:
[Equation]
[Equation]
Because the exponential map in the Poincaré ball model is a bijective function, the parallel transported gyrovectors w and z can be regarded as the exponential mapped tangent vectors v ∈ T x B nc rooted at x and u ∈ T y B nc rooted at y , respectively, that is,
[Equation]
With Equations 17 and 18 and the properties of the Möbius gyration described in Appendix B, a succinct expression of the tangent parallel transport P c x → y : T x B nc
→ T y B nc can be obtained as follows:
[Equation]
Note that, in a special case in which x = 0 and v ∈ T
B nc , this equation is simpliﬁed as follows:
[Equation]
[Equation]
One can conﬁrm that Equation 26 deserves to be called a parallel transport in terms of the differential or Riemannian geometry by checking the covariant derivative associated with the Levi-Civita connection of P c x → y along a tangent vector ﬁeld ˙ γ ( t ) on a smooth curve γ ( t ) from x to y vanishes to 0 .
D S UPPLEMENTAL PROOFS FOR PROPOSED METHODS D. F INAL DEFORMATION OF THE PROPOSED UNIDIRECTIONAL P OINCARÉ MLR
Proof.
First, we clarify the relation between the Poincaré hyperplane
˜
H c a , p , described in Appendix
C.2.2, and the variants ¯ H c a ,r introduced in Section 3.1:
[Equation]
We then start the derivation of Equation 6 from the variables a k and q a k ,r k described in Section 3.1.
Following Equation 28 and the concept of the distance from a point to a Poincaré hyperplane described in Equation 23, the generalized MLR score function v k in Equation 3 can be written as follows:
[Equation]
With Equation
[Equation]
we obtain
[Equation]
Therefore
[Equation]
we can expand the term inside the sinh − 1 function in Equation 29 in the following manner:
[Equation]
With Equations 5 and
[Equation]
the term in the outer brackets in Equation 33 can be further expanded into the form using r k and z k described in Section 3.1:
[Equation]
In addition
[Equation]
we can also expand the term outside the sinh − 1 function in Equation 29 using Equations 5 and 17 as follows:
[Equation]
Combining Equations 29, 33,
[Equation]
and
[Equation]
we ﬁnally conclude the proof through the following:
[Equation]
D. C ONVERGENCE PROOF OF UNIDIRECTIONAL P OINCARÉ MLR TO E UCLIDEAN MLR
Proof.
For the intended proof
[Equation]
we ﬁrst introduce the following proposition:
Proposition 1.
For x (cid:54) = 0 , sinh( x ) over x converges to 1 in the limit x → 0 :
[Equation]
Proof.
The result can be obtained based on the deﬁnition of the differentiation of a scalar function:
[Equation]
From Proposition
[Equation]
we derive the following two propositions
[Equation]
Proposition 2.
For t ∈ R , x (cid:54) = 0 , sinh( tx ) over x converges to t in the limit x → 0 :
[Equation]
Proof.
We divide this proof into two cases:
[Equation]
Proposition 3.
For t ∈ R , x (cid:54) = 0 , sinh − 1 ( tx ) over x converges to t in the limit x → 0 :
[Equation]
Proof.
We can directly utilize Proposition 1 as follows:
[Equation]
Proposition 1 )
[Equation] [Equation]
With Propositions 2 and
[Equation]
we can now take the limit of Equation 6 as follows:
[Equation]
Moreover, with Equation
[Equation]
we can conﬁrm that z k matches a k in the limit c → 0 :
[Equation]
Combining it with Equations 2 and
[Equation]
we ﬁnally conclude the proof as follows:
[Equation]
b k
[Equation]
r
[Equation]
k (cid:107) .
[Equation]
Here, the factor 4 is derived from the squared conformal factor ( λ 0 x ) 2 degenerating into a constant value.
This corresponds to the fact that the Poincaré ball model B nc converges to the Euclidean space R n in the limit c → 0 except for the same multiplier lim c → 0 ( λ c x ) 2 = 4 owing to its metric tensor.
D. P ROOF OF THE PROPERTIES OF OUTPUT COORDINATES OF P OINCARÉ FC LAYER
Proof.
To check the properties of the Poincaré FC layer described in Section 3.
[Equation]
we ﬁrst clarify the
Poincaré hyperplane containing the origin and orthogonal to the k -th axis in B mc .
The k -th axis is a geodesic passing through the origin and any point on it except the origin has a non-zero element in only the k -th coordinates.
Therefore, an arbitrary point x ∈ B m c along the k -th axis can be written as follows:
[Equation]
(53) which is as intuitive as in a Euclidean space.
Speciﬁcally, r = 0 represents the origin.
We can then easily describe the intended Poincaré hyperplane as follows:
Deﬁnition 1.
(Poincaré hyperplane containing the origin and orthogonal to the k -th axis)
[Equation] [Equation]
which is also intuitively obtained.
With Deﬁnition 1, the preparation for constructing y in Equation 7 is complete.
Derivation of y .
Let x ∈ B nc and y
=
( y 1 , y 2 , . . .
, y m ) (cid:62) ∈ B mc be the input and output of the
Poincaré FC layer, respectively.
Below, we start the proof with the score functions v k ( x ) for ∀ k
= { 1 , 2 , . .
. , m } already obtained in the same way as in Equation 6.
To endow y the properties described in Section 3.2, i .
e ., the signed distance from y to each Poincaré hyperplane containing the origin and orthogonal to the k -th axis is equal to v k ( x ) , we generate a simultaneous equation for ∀ k as follows:
[Equation]
[Equation]
With Equations 54 and 28 and the notion of the distance from a point to a Poincaré hyperplane described in Equation 23, these equations are expanded as follows:
[Equation]
[Equation]
Therefore, we obtain the following notation of the coordinates:
[Equation]
When considering the Euclidean norm of y using Equation 57, the equation for (cid:107) y (cid:107) can be derived as follows:
[Equation]
This can be succinctly rewritten as
[Equation] [Equation]
By solving this quadratic equation
[Equation]
the closed form of (cid:107) y (cid:107) is obtained through the following:
[Equation]
Substituting Equations 59 and 60 for Equation 57 leads to Equation 7 in the notation of the coordinates:
[Equation]
Conﬁrmation of the existence of y .
Finally
[Equation]
we conclude the proof by checking that y is always within the domain of the Poincaré ball B m c = { y ∈ R m | c (cid:107) y (cid:107) 2 < 1 } :
[Equation]
D. P ROOF OF THE PROPERTIES OF P OINCARÉ β - SPLIT AND β - CONCATENATION
In this section, we prove the properties of the Poincaré β -split
and the Poincaré β -concatenation described in Section 3.3.
The Poincaré ball model is different from Euclidean neural networks, on the simple calculation of the expected value and the variance of a particular value related to a feature vector or weight matrix owing to the linearity in their operations.
In the Poincaré ball model, calculating such values without any postulate for the probabilistic distribution that the feature gyrovectors or tangent vectors follow is difﬁcult owing to the nonlinear transformations in the exponential and logarithmic maps.
Thus, we ﬁrst make the following naive assumption:
Assumption 1.
Each coordinate of an n -dimensional tangent vector in T 0 B nc follows a normal distribution centered at zero with a certain variance σ 2 n c .
The reasons why we assume the distribution on the tangent space rather than on the Poincaré ball model itself are as follows:
It is improper to assume a continuous and smooth distribution onto the space with an upperbounded radius because there must be no probability density on or outside the boundary.
The rough idea of discontinuing such probabilities outside the domain of the Poincaré ball and discretely taking only the inside into account seems to lack rationality.
One simple way to avoid the above issue is to apply a uniform distribution from zero to the ball radius based on the norm of the gyrovector.
However, there is no guarantee that such constancy in the distribution can be realized on a complexly curved geometric structure of the Poincaré ball model.
Conversely, a tangent space is a linear space that is attached to the manifold and can be treated as an ordinary vector space.
The Poincaré ball model is conformal to the Euclidean space,
i
e ., preserving the same angles, and at the origin, the gyrovectors having the same norms are projected onto the tangent vectors which also have the same norms with their angles unchanged.
In Euclidean neural networks, the normal distribution is one of the most popularly considered priors.
The multivariate normal distribution is occasionally approximated
as an independent and identically distributed distribution for easier calculation.
Because the Poincaré β -split and the Poincaré β -concatenation are inverse functions to each other, it is sufﬁcient to prove the properties of either one of these operations.
Here, we show a proof for the
Poincaré β -concatenation.
Recalling that β n = B( n 2 , 12 ) and considering the following:
Poincaré β -concatenation.
The input gyrovectors { x i ∈ B n i c } Ni =1 are ﬁrst scaled by certain coefﬁcients and concatenated in the tangent space, and then projected back to the Poincaré ball as follows:
[Equation]
Proof.
At ﬁrst, we consider the expected value of the norm of each tangent vector v i , which is the target of the Poincaré β -concatenation.
Because the value t i :=
c
[Equation] [Equation]
follows a χ 2 distribution based on Assumption 1, the expected value of (cid:107) v i (cid:107) can be obtained as follows:
[Equation] [Equation]
Therefore, when the norm of each input tangent vector v i is kept the same by the former part of neural networks before applying this operation, the standard deviation σ n i must be expressed as follows:
[Equation]
In addition, using Equation 63, the squared norm of the Poincaré β -concatenated tangent vector v is obtained as follows:
[Equation]
This leads the value t :
= c (cid:107) v (cid:107) 2 σ 2 n , where σ n = Cβ n , which is expressed as follows:
[Equation]
Here, t also follows a χ 2 distribution, and the expected value of the norm of v is obtained as follows:
[Equation]
which is the same as the norms of the input tangent vectors.
This indicates that each coordinate of v follows a normal distribution centered at zero with a variance σ 2 n c , satisfying the Assumption 1.
Based on the results above, the expected value of the norm of each input gyrovector x i is expressed by the following:
[Equation]
Note that, for the calculation between Equations 75 and
[Equation]
we utilize the Taylor series expansion of tanh for a real value.
Furthermore, considering the Laurent series expansion at inﬁnity
[Equation]
we can obtain the following expressions:
[Equation]
Therefore, in the general cases in which n i (cid:29)
[Equation]
we can obtain the following approximation:
[Equation]
Note that, for the calculation between Equations 84 and 85, we utilize Stirling’s approximation, i .
e ., Γ( z ) (cid:39) (cid:113)
2 πz (cid:0) ze (cid:1) z .
In addition, we utilize the deﬁnition of Napier’s constant for the approximation between Equations 88 and 89, i .
e ., lim x →∞ (1 + 1 x ) x
= e .
Combining Equations 79 and 90, the expected value of (cid:107) x i (cid:107) can be approximately expressed by the following:
[Equation]
In the same way, the expected value of the Poincaré β -concatenated gyrovector x is obtained by the following:
[Equation]
[Equation] [Equation]
which concludes the proof.
D. T HE M ÖBIUS GYROMIDPOINT AND THE E INSTEIN GYROMIDPOINT
Einstein gyromidpoint.
In the Beltrami-Klein model, the midpoint ¯ n ∈ K nc among { n i ∈ K nc } Ni = and the non-negative scalar weights { ν i ∈ R + } Ni =1 is given as follows:
[Equation]
, where γ
[Equation]
This operation is called the Einstein gyromidpoint (Ungar, 2009).
Based on the above, we prove the equivalence of the Möbius gyromidpoint and Einstein gyromidpoint.
Proof.
Let the points { b i ∈ B nc } Ni =1 correspond to { n i ∈ K nc } Ni =1 , respectively, i .
e .,
b i is a projection of n i to the Poincaré ball model using Equation 11.
From Equations 12 and 96, we obtain the following:
[Equation]
Substituting Equations 12 and 97 for Equation 96 leads to the representation of the Einstein midpoint using the coordinates in the Poincaré ball model:
[Equation]
Therefore, the point ¯ b ∈ B nc , which is a projection of ¯ n to the Poincaré ball model using Equation 11, is expressed in the following manner:
[Equation]
This concludes the proof.
D. M ÖBIUS GYROMIDPOINT AND CENTROID OF SQUARED L ORENTZIAN DISTANCE
Weighted centroid in the hyperboloid model (Law et al., 2019) .
With a Lorentzian norm |(cid:107) x (cid:107) L | = (cid:112) |(cid:104) x , x (cid:105) L | =
(cid:112) |(cid:107) x (cid:107) 2 L | for x ∈ R n + , the center of mass ¯ h ∈ H nc among { h i =
( z i , k (cid:62) i ) (cid:62) ∈
H nc } Ni =1 and the non-negative scalar weights { ν i ∈ R + } Ni =1 is given as follows:
[Equation]
This is based on the minimization problem of the weighted sum of squared Lorentzian distances expressed as follows:
[Equation]
In the following, we prove the equivalence of the Möbius gyromidpoint and the weighted centroid in the hyperboloid model.
Proof.
Expanding Equation 100 with the coordinates, we obtain the following:
[Equation]
The point ¯ b ∈ B nc , which is a projection of ¯ h to the Poincaré ball model using Equation
[Equation]
is expressed in the following manner:
[Equation]
Dividing both the numerator and denominator by (cid:80) i
ν
i z
i , this can be rewritten as follows:
[Equation]
where b :=
[Equation]
c
N (cid:88)
i
[Equation] [Equation]
Next, considering the points { b i ∈ B nc } Ni =1 , which also correspond to { h i } Ni =1 , respectively, we can transform the expression of b into an expression with only the coordinates in the Poincaré ball model:
[Equation]
This concludes the proof.
D. M ÖBIUS GYROMIDPOINT AS A SOLUTION OF THE MINIMIZATION PROBLEM
The discovery of the equivalence between the weighted centroid in the hyperboloid model and the
Möbius gyromidpoint enables us to discuss what the Möbius gyromidpoint is a minimizer of.
In the following, we prove that the Möbius gyromidpoint can be regarded as a minimizer of the weighted sum of calibrated squared gyrometrics.
Theorem 2.
The Möbius gyromidpoint is a solution of the minimization problem of the weighted sum of calibrated squared gyrometrics, which is expressed as follows:
[Equation]
[Equation]
Each (cid:107) (cid:9) c
¯ b ⊕ c b i (cid:107) indicates the norm of the respective gyrovector
b i viewed from the Möbius gyromidpoint ¯ b , which equals the gyrodistance of ¯ b to b i and is also called a gyrometric (Ungar, 2009).
In addition, each λ c (cid:9) c ¯ b ⊕ c b i is a conformal factor of the metric tensor of the Poincaré ball model for such a gyrovector.
Therefore, the minimization objective in Equation 106 can be interpreted as the weighted sum of squared gyrometrics, each of which is calibrated by a scaling factor at the respective point.
Proof.
Let the point ¯ b ∈ B nc be a projection of the weighted centroid ¯ h ∈ H nc .
With Equationand the notation of Equation
[Equation]
we obtain the following straightforward expression:
[Equation]
Expanding Equation 107 with the coordinates
[Equation]
we obtain the following:
[Equation]
Considering the points { b i ∈ B nc } Ni =
[Equation]
which correspond to { h i } Ni =
[Equation]
respectively
[Equation]
we can transform
Equation 108 into an expression with only the coordinates in the Poincaré ball model:
[Equation]
This concludes the proof.
D. W EIGHT GENERALIZATION OF THE M ÖBIUS GYROMIDPOINT
As mentioned in Section 3.5, we extend the condition of the weights of the Möbius gyromidpoint to all real values { ν i ∈ R } Ni =1 by regarding a negative weight as an additive inverse operation, that is, regarding any pair ( ν i , b i ) as ( | ν i | , sign( ν i ) b i ) :
[Equation]
E I MPLEMENTATION D ETAILS E. P ARAMETER INITIALIZATION
Unidirectional Poincaré MLR.
When the dimensions of the input gyrovector is n , each element of the weight parameter Z is initialized by a normal distribution centered at zero with a standard deviation n − 1 2 .
The bias parameter r is initialized as a zero vector.
Poincaré FC layer.
When the dimensions of the input gyrovector and the output gyrovector are n and m , respectively, each element of the weight parameter Z is initialized by a normal distribution centered at zero with a standard deviation (2 nm ) − 12 .
The bias parameter r is initialized as a zero vector.
Poincaré convolutional layer.
When the dimensions of the input gyrovector and the output gyrovector are n and m , respectively, and the total kernel size is K , each element of the weight parameter
Z is initialized by a normal distribution centered at zero with a standard deviation (2 nKm ) − 12 .
The bias parameter r is initialized as a zero vector.
Embedding on the Poincaré ball model.
As mentioned in Ganea et al.
(2018a)
, we conﬁrmed the tendency of the parameters in the Poincaré ball model to adjust their angles at the ﬁrst phase of the training before increasing their norms.
In addition, we consider that, due to the exponentially growing distance metric of the hyperbolic space, the farther a gyrovector parameter is placed from the origin, the more costly it moves such a point to another point through the optimization.
Therefore, the embedding parameters on the Poincaré ball model should be initialized with a particular small gain (cid:15) E , given as a hyperparameter, aiming to accelerate such an adjustment and make the later optimization smooth.
We set the value (cid:15) E to be 10 − 2 in the experiment in Section 4.3.
E.
H YPERPARAMETERS OF THE EXPERIMENT IN S ECTION 4.
Optimization.
We used the Riemannian Adam optimizer with β 1 = 0 .
9 , β 2 = 0 .
999 and (cid:15) =
10 −for both of the Euclidean and our hyperbolic architectures.
The learning rate η was set to 10 − 3 .
E. H YPERPARAMETERS OF THE EXPERIMENT IN S ECTION 4.
Model architectures.
Let D be the dimension of the source and target token embeddings.
Each model for the experiment in Section 4.3 has the encoder and decoder, both of which are composed of ﬁve convolutional layers with a kernel size of three and a channel size of D , ﬁve convolutional layers with a kernel size of three and a channel size of 2 D , and two convolutional layers with a kernel size of one and a channel size of 4 D .
The output feature maps of the last convolutional layer in the encoder are projected into D -dimensional feature maps.
They are utilized as the key for the encoder-decoder attentions.
Likewise, the output feature maps of the last convolutional layer in the decoder are projected into D -dimensional feature maps for the ﬁnal token classiﬁcation.
Training.
In each iteration of the training phase, we fed each model a mini-batch containing approximately 10,000 tokens at most.
In this setting, the batch size, or the number of the sentence pairs in a mini-batch, dynamically changes.
As a loss function, we utilized the cross entropy function with a label smoothing of 0.1.
Optimization.
We used the Riemannian Adam optimizer with β 1 = 0 .
9 , β 2 = 0 .
98 and (cid:15) = 10 −for both of the Euclidean and our hyperbolic architectures.
For the scheduling of the learning rate η , we linearly increased the learning rate for the ﬁrst 4000 iterations as a warm-up, and utilized the inverse square root decay with respect to the number of iterations t thereafter as η = ( Dt ) − 12 .