U NIT B ALL M ODEL FOR E MBEDDING H IERARCHICAL S TRUCTURES IN THE C OMPLEX H YPERBOLIC S PACE
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning the representation of data with hierarchical structures in the hyperbolic space attracts increasing attention in recent years.
Due to the constant negative curvature, the hyperbolic space resembles tree metrics and captures the tree-like properties naturally, which enables the hyperbolic embeddings to improve over traditional Euclidean models.
However, most real-world hierarchically structured data such as taxonomies and multitree networks have varying local structures and they are not trees, thus they do not ubiquitously match the constant curvature property of the hyperbolic space.
To address this limitation of hyperbolic embeddings, we explore the complex hyperbolic space, which has the variable negative curvature, for representation learning.
Speciﬁcally, we propose to learn the embeddings of hierarchically structured data in the unit ball model of the complex hyperbolic space.
The unit ball model based embeddings have a more powerful representation capacity to capture a variety of hierarchical structures.
Through experiments on synthetic and real-world data, we show that our approach improves over the hyperbolic embedding models signiﬁcantly.
We also explore the competence of complex hyperbolic geometry on the multitree structure and 1 - N structure.
I NTRODUCTION
Representation learning of data with hierarchical structures is an important machine learning task with many applications, such as taxonomy induction (Fu et al., 2014) and hypernymy detection (Shwartz et al., 2016).
In recent years, the hyperbolic embeddings (Nickel and Kiela, 2017; 2018) have been proposed to improve the traditional Euclidean embedding models.
The constant negative curvature of the hyperbolic space produces a manifestation that the hyperbolic space can be regarded as a continuous approximation to trees (Krioukov et al., 2010).
The hyperbolic space is capable of embedding any ﬁnite tree while preserving the distances approximately (Gromov, 1987).
However, most real-world hierarchical data do not belong to tree structures since they can have varying local structures while being tree-like globally.
For example, the taxonomies such as WordNet (Miller, 1995) and YAGO (Suchanek et al., 2007) contain many 1 - N ( 1 child links to multiple parents) cases and multitree structures (Griggs et al., 2012), which are much more complicated than the tree structure.
In consequence, the hyperbolic space which resembles tree metrics has limitations on capturing the general hierarchically structured data.
To address the challenge, in this paper, we propose a new approach to learning the embeddings of hierarchically structured data.
Speciﬁcally, we embed the hierarchical data into the unit ball model of the complex hyperbolic space.
The unit ball model is a projective geometry based model to identify the complex hyperbolic space.
One of the main differences between the complex and the real hyperbolic space is that the curvature is non-constant in the complex hyperbolic space.
In practice, the variable negative curvature makes the complex hyperbolic space more ﬂexible in handling varying structures while the tree-like properties are still retained.
For empirical evaluation, we evaluate different geometrical embedding models on various hierarchically structured data, including synthetic graphs and real-world data.
The experimental results demonstrate the advantages of our approach.
In addition, we investigate two speciﬁc structures in which complex hyperbolic geometry shows outstanding performances, namely the multitree structure and 1 - N structure, which are highly common and typical in real-world data.
To summarize, our work has the following main contributions: 1.
We present a novel embedding approach based on the complex hyperbolic geometry to handle data with complicated and various hierarchical structures.
To the best of our knowledge, our work is the ﬁrst to propose complex hyperbolic embeddings.
We introduce the embedding algorithm in the unit ball model of the complex hyperbolic space.
We formulate the learning and Riemannian optimization in the unit ball model.
We evaluate our approach with experiments on an extensive range of synthetic and real-world data and show the remarkable improvements of our approach.
R ELATED W ORK
Hyperbolic embeddings.
Hyperbolic embedding methods have become the leading approach for representation learning of hierarchical structures.
Nickel and Kiela (2017) learned the representations of hierarchical graphs in the Poincar´e ball model of the hyperbolic space and outperformed the
Euclidean embedding methods for taxonomies.
The Poincar´e embedding model was then improved by follow-up works on hyperbolic emebddings (Ganea et al., 2018a; Nickel and Kiela, 2018).
These methods learned the hyperbolic embeddings by Riemannian optimization (Bonnabel, 2013), which was further improved by the Riemannian adaptive optimization (B´ecigneul and Ganea, 2019).
Another branch of study (Sala et al., 2018; Sonthalia and Gilbert, 2020) learned the hyperbolic embeddings through combinatorial construction.
Instead of optimizing the soft-ranking loss by
Riemannian optimization as in (Nickel and Kiela, 2017; 2018), the construction-based methods minimize the reconstruction distortion by combinatorial construction.
However, both the optimizationbased and construction-based hyperbolic embeddings suffer from the limitation in hierarchical graphs with varying local structures.
To tackle the challenge, Gu et al.
(2019) extended the construction-based method by jointly learning the curvature and the embeddings of data in a product manifold.
Although it can provide a better representation than a single space with constant curvature, it is impractical to search for the best manifold combination among enormous combinations for each new structure.
Motivated by the promising results of previous works, extensions to the multi-relational graph hyperbolic embeddings (Balazevic et al., 2019; Chami et al., 2020; Sun et al., 2020) and hyperbolic neural networks (Ganea et al., 2018b; G¨ulc¸ehre et al., 2019; Liu et al., 2019; Chami et al., 2019;
Zhu et al., 2020; Dai et al., 2021a;
Shimizu et al., 2021) were explored.
Notably, (Chami et al., 2019; 2020) leverages the trainable curvature to compensate for the disparity between the actual data structures and the constant-curvature hyperbolic space, where each layer in the graph neural network or each relation in the multi-relational graph has its own curvature parameterization.
Since we only focus on the single-relation graph embeddings and taxonomy embeddings in this work, we do not evaluate the multi-relational knowledge graph embedding models or the neural networks in our tasks.
The hyperbolic learning also inspired other research tasks and applications, such as classiﬁcation (Cho et al., 2019), image reconstruction (Skopek et al., 2020), text generation (Dai et al., 2021b), etc.
Complex embeddings.
The traditional knowledge graph embeddings were learned in the real
Euclidean space (Nickel et al., 2011; Bordes et al., 2013; Yang et al., 2015) and were used for knowledge graph inference and reasoning.
In recent years, several works suggested utilizing the complex Euclidean space for inferring more relation patterns, such as ComplEx (Trouillon et al., 2016) and RotatE (Sun et al., 2019).
These models have been demonstrated to be effective in knowledge graph embeddings.
The success of the complex embeddings reveals the potential of the complex space and inspires us to explore the complex hyperbolic space.
P RELIMINARIES 3. H YPERBOLIC G EOMETRY
Hyperbolic space 1 is a homogeneous space with constant negative curvature.
In the hyperbolic space H n
R ( K ) of dimension n and curvature K , the volume of a ball grows exponentially with its radius ρ :
[Equation]
Contrastively, in the Euclidean space E n , the curvature is 0 and the volume of a ball grows polynomially with its radius:
[Equation]
The exponential volume growth rate enables the hyperbolic space to have powerful representation capability for tree structures since the number of nodes grows exponentially with the depth in a tree, while the Euclidean space is too ﬂat and narrow to embed trees.
3. C OMPLEX H YPERBOLIC G EOMETRY
Complex hyperbolic space is a homogeneous space of variable negative curvature.
Its ambient
Hermitian vector space C n, 1 is the complex Euclidean space C n +1 endowed with some Hermitian form (cid:104)(cid:104) z , w (cid:105)(cid:105) , where z , w ∈ C n +1 .
Then the Hermitian space C n, 1 can be divided into three subsets:
V − = { z ∈ C n, 1 |(cid:104)(cid:104) z , z (cid:105)(cid:105) < 0 } , V
= { z ∈ C n, 1 − { 0 }|(cid:104)(cid:104) z , z (cid:105)(cid:105) = 0 } , and V + = { z ∈ C n, 1 |(cid:104)(cid:104) z , z (cid:105)(cid:105) > 0 } .
Let P be a projection map P : C n, 1 − { z n +1 = 0 } → C n , i.e.,
P :   z 1 .
z n +   (cid:55)→   z 1 /z n +1 .
z n /z n + 
 , where z n +1 (cid:54) = 0 .
Then the complex hyperbolic space H n C and its boundary ∂ H n C are deﬁned using the projectivization:
[Equation]
The curvature of the complex hyperbolic space is summarized by (Goldman, 1999) as follows:
Theorem 1.
The curvature is not constant in H n C .
It is pinched between − 1 (in the directions of complex projective lines) and − 1 / 4 (in the directions of totally real planes).
We leave the full proof in Appendix B.
The non-constant curvature, which we expect to be favorable for embedding various hierarchical structures, is one of the main differences between H n C and H n R .
The complex hyperbolic space also has the tree-like exponential volume growth property.
The volume of a ball with radius ρ in H n C is given by
[Equation]
where σ 2 n − 1 = 2 π n /n !
is the Euclidean volume of the unit sphere S 2 n − 1 ∈ C n .
From the properties of the complex hyperbolic geometry, we expect that the complex hyperbolic space can naturally handle data with diverse local structures in virtue of the variable curvature as presented in Theorem 1 while preserving the tree-like properties as shown in Eq.
From this section, we see that complex hyperbolic geometry and hyperbolic geometry are typically of different characteristics.
The n -dimensional ( n -d) complex hyperbolic space is not simply the 2 n -d hyperbolic space or the product of two n -d hyperbolic spaces.
This implies that our complex hyperbolic embedding model is intrinsically different from the hyperbolic embedding methods (Nickel and Kiela, 2017; 2018) or the product manifold embeddings (Gu et al., 2019).
U NIT B ALL E MBEDDINGS
We propose to embed the hierarchically structured data into the unit ball model of the complex hyperbolic space.
In this section, we introduce our approach in detail.
4. T HE U NIT B ALL M ODEL
The unit ball model is one model used to identify the complex hyperbolic space, which can be derived via the projective geometry (Goldman, 1999).
We now provide the necessary derivation.
Take the Hermitian form of C n, 1 in Section 3.2 to be a standard Hermitian form:
[Equation]
where w is the conjugate of w .
Take z n +1 = 1 in the projection map P in Eq.
Then from Eq.
[Equation]
we can derive the formula of the unit ball model:
[Equation]
The metric on B n C is Bergman metric, which takes the formula below in 2 -d case:
[Equation]
The distance function on B n C is given by
[Equation]
Note that there are other choices of the Hermitian form (cid:104)(cid:104) z , w (cid:105)(cid:105) , which corresponds to other models of complex hyperbolic geometry, such as the Siegel domain model.
We choose the unit ball model for the relatively simple formula as well as convenient computations of the metric and distance function.
4. E MBEDDINGS IN THE U NIT B ALL M ODEL
Given the hierarchical data containing a set of nodes X = { x p } mp =1 and a set of edges E = { ( x p , x q ) | x p , x q ∈ X } , we aim to learn the embeddings of the nodes Z = { z p } mp =1 , where z p ∈ B n C .
The objective of the embeddings is to recover the structures of input data, including the distances between the nodes as well as the partial order in the hierarchies.
Here we adopt the soft ranking loss used in the Poincar´e ball embeddings (Nickel and Kiela, 2017) and the hyperboloid embeddings (Nickel and Kiela, 2018), which aims at preserving the hierarchical relationships among nodes:
[Equation]
10) where N ( x p ) =
{ x k : ( x p , x k ) / ∈ E } ∪ { x p } is the set of negative examples for x p together with x p .
d B n C is the distance function in the unit ball model given in Eq.
The minimization of L makes the connected nodes closer in the embedding space than those with no observed edges.
The learning process implicitly aligns the geometric structures of the embedding space and the underlying graph structures of data since the loss function aims at preserving the hierarchical relationships among nodes while the underlying graph structures are reﬂected by the hierarchical relationships.
We learn the embeddings in the unit ball model, where the variable negative curvature of the complex hyperbolic space provides the capacity to deal with more varying structures.
The experiments in Section 5 exhibit that the unit ball model learns the high-quality embeddings and captures the various hierarchical structures.
4. R IEMANNIAN O PTIMIZATION IN THE U NIT B ALL M ODEL
We learn the embeddings Z = { z p } mp =1 through solving the optimization problem with constraint:
[Equation]
For the optimization problems in Riemannian manifolds, Bonnabel (2013) presented the Riemannian stochastic gradient descent (RSGD) algorithm, which we employ to optimize Eq.
To update an embedding z ∈ B n C , 3 we need to obtain its Riemannian gradient ∇ R .
Speciﬁcally, the embedding is updated at the t -th iteration by z ( t ) ← z ( t − 1) − η ( t ) ∇ R L ( z ) , where η ( t ) is the learning rate at the t -th iteration and ∇ R L ( z ) is the Riemannian gradient of L ( z ) .
The Riemannian gradient ∇ R can be derived from rescaling the Euclidean gradient ∇ E with the inverse of the metric tensor ds 2 in Eq.
Apply the chain rule of differential functions and we have:
[Equation]
∂L
[Equation]
z ) ∂d B n C ( z , w ) is trivial to compute from Eq.
In practical training, we implement and compute the complex hyperbolic embedding as its real part and imaginary part, i.e., z = x
+ i y , where i represents the imaginary unit , i.e., i 2 = − 1 .
In order to get the gradient of the distance function ∇ E d B n C ( z , w ) in Eq.
(12), we get the partial derivative with regard to the real part and the imaginary part, i.e., ∇ E d B n C ( z , w ) =
∂d B n C ( z , w ) ∂
x + i ∂d B n C ( z , w ) ∂ y .
The partial derivatives of the unit ball model distance take the following formulas:
[Equation] [Equation]
where p = cosh( d B n C ( z , w )) .
Re ( · ) and Im ( · ) denote the real and the imaginary part respectively.
The full derivation of Eqs.
(13) and (14) is given in Appendix C.
Since the embedding z should be constrained within the unit ball model, we apply the same projection strategy as (Nickel and Kiela, 2017) via a small constant ε :
[Equation] [Equation]
To sum up, the update of z at the t -th iteration is
[Equation] [Equation]
The RSGD steps of the unit ball embeddings are presented in Algorithm 1.
E XPERIMENTS
In experiments, we evaluate the performances of our approach and baselines on various hierarchical structures, including synthetic graphs and real-world data.
We focus on the graph reconstruction task and the link prediction task.
The main results are reported in this section.
For more experiments, please refer to Appendix F.
5. E XPERIMENTAL S ETTINGS 5.1. D ATA
We use synthetic and real-world data that exhibit underlying hierarchical structures to evaluate our approach.
The public links or generator package links of the data are given in Appendix F.1.
Synthetic.
We generate various balanced trees and compressed graphs using NetworkX package (Hagberg et al., 2008).
For balanced trees , we generate the balanced tree with degree r and depth h .
For
[Table]
compressed graphs , we generate k random trees on m nodes and then aggregate their edges to form a graph.
Some examples of the synthetic data are given in Appendix F.1.
Xiphophorus.
The Xiphophorus is a multitree dataset from (Cui et al., 2013), which containstrees representing mrbayes consensus trees inferred for different genomic regions on 26 Xiphophorus ﬁshes.
We use the saved MultiTree object in toytree package.
ICD10.
The 10 -th revision of International Statistical Classiﬁcation of Diseases and Related Health
Problems (ICD10) (Br¨amer, 1988) is a medical classiﬁcation list provided by the World Health
Organization.
We construct its full transitive closure as the ICD10 dataset.
YAGO3-wikiObjects.
YAGO3 (Mahdisoltani et al., 2015) is a huge semantic knowledge base.
It provides a taxonomy derived from Wikipedia and WordNet.
We extract the Wikipedia concepts and entities that are descendants of (cid:104) wikicat Objects (cid:105) as well as the hypernymy edges among them.
We compute the transitive closure of the sampled taxonomy to construct the YAGO3-wikiObjects dataset.
WordNet-noun.
WordNet (Miller, 1995) is a large lexical database.
The hypernymy relation among all nouns forms a noun hierarchy.
We use its full transitive closure as the WordNet-noun dataset.
For the multitree (Xiphophorus), we use the full dataset as the training set and the edges containing the leaf nodes as the test set.
For each real-world taxonomy (ICD10, YAGO3-wikiObjects, WordNetnoun), we randomly split the edges into train-validation-test sets with the ratio 90% : 5% : 5% .
We make sure that any node in the validation and test sets must occur in the training set since otherwise, it cannot be predicted.
But the edges in the validation and test sets do not occur in the training set since they are disjoint.
We provide the statistics of the real-world datasets in Table 1.
The
Gromov’s δ -hyperbolicity (Gromov, 1987) measures the tree-likeness of graphs (refer to Appendix D for deﬁnition).
The lower δ corresponds to the more tree-like graph and trees have 0 δ -hyperbolicity.
5.1. T ASKS
We evaluate the following two tasks:
Graph reconstruction : we train the embeddings of the full data and then reconstruct it from the embeddings.
The task evaluates representation capacity.
Link prediction : we train the embeddings on the training set and predict the edges in the test set.
The task evaluates generalization performance.
5.1. B ASELINES
We compare our approach UnitBall to the following methods: the sate-of-the-art combinatorial construction-based hyperbolic embedding method TreeRep (Sonthalia and Gilbert, 2020), the optimization-based hyperbolic embeddings in the Poincar´e ball model (Nickel and Kiela, 2017) and the Hyperboloid model (Nickel and Kiela, 2018), the simple Euclidean embedding model.
Euclidean, Poincar´e, Hyperboloid, and our approach UnitBall use the same loss function but learn in the different geometrical spaces.
Therefore, the comparisons reveal the capacities of different geometrical models in different spaces.
For the baselines, we use their released codes to train the embeddings.
For all methods, the hyperparameters are tuned on each validation set for the link prediction task and on balanced tree-(15,3) for the graph reconstruction task.
The hardware information is given in Appendix F.2 and the hyperparameters are listed in Appendix F.3.
In all experiments, we report the mean results over 5 running executions.
The code of our approach will be publicly available after the publication of the paper.
[Figure]
5.1. E VALUATION
We use the mean average precision ( MAP ), mean reciprocal rank ( MRR ), and Hits@N as our evaluation metrics, which are widely used for evaluating ranking and link prediction.
The details of prediction steps and the evaluation metrics are given in Appendix F.4.
The n -d complex hyperbolic embeddings have around double parameters of the n -d real embeddings since the n -d complex hyperbolic vectors have n -d real part and n -d imaginary part.
For a fair comparison, in each experimental setting, we compare our n -d complex hyperbolic embeddings of
UnitBall against the 2 n -d embeddings of the baselines.
The results will also demonstrate that the n -d complex hyperbolic space is not simply the 2 n -d hyperbolic space, they have different capacities.
5. G RAPH R ECONSTRUCTION 5.2. R ESULTS ON C OMPRESSED G RAPHS
To illustrate the beneﬁts of UnitBall on varying hierarchical structures, we evaluate on the synthetic compressed graphs.
The compressed graphs have local tree structures while being much more complicated than trees.
Each compressed graph- ( m, k ) consists of m nodes and is aggregated from k random trees on the m nodes.
The bigger k corresponds to the denser and noisier graph.
Figure 1 depicts that the graph reconstruction results drop down with the increase of m and k , which represents the increase of graph scale and denseness respectively.
Remarkably, UnitBall outperforms all baselines on the challenging data, showing that UnitBall handles the noisy locally tree-like structures better.
The construction-based method TreeRep learns a tree structure from the data as an intermediate step and then embeds the learned trees into the hyperbolic space using
Sarkar’s construction (Sarkar, 2011).
TreeRep has comparable results with other methods when ( m, k ) =
(500 , 1) since when k = 1 , the graph is exactly a tree, i.e., δ = 0 .
However, when k >and δ > 0 , the data metrics deviate from tree metrics, in which case it does not help much to learn a tree structure from the data as an intermediate step.
5.2. R ESULTS ON M ULTITREE S TRUCTURE
In this section, we compare the performances of the hyperbolic models and UnitBall on Xiphophorus.
Xiphophorus is a multitree dataset that is formed of 160 mrbayes consensus trees on 26 Xiphophorus ﬁshes.
Its cloud tree plot is in Figure 2.
The multitree structure is widely used to represent multiple overlapping taxonomies over the same ground set (please refer to Appendix E for details).
We reconstruct the edges containing the leaf nodes on Xiphophorus since the leaf links have the practical taxonomic meaning.
The results are reported in the last column of Table 4.
The results show that the complex hyperbolic geometry has a stronger ability to represent the multitree structure.
X .
m a li
n c he_ CH I C
X .
b i r c h m ann i
_ G A RC X .
c o r t e z
i
X .
ne z ahua c
o y
o t l
X .
m on t e z u m ae
X .
n i g r
en s i s
X . m
u l t ili nea t u s
X .
p
y g m
aeu s
X .
c on t i nen s
X .
m a y ae X .
a l v a r e z i X .
he ll e r ii X .
s i
gnu m
[Figure]
m on t
i c
o l u s
[Figure]
l
[Figure]
en
[Figure]
ae
[Figure]
X .
m e y e r
i
X .
go r don
i
X .
c ou c h i anu s
X .
v a r i a t u s
X .
e v e l y nae
X .
m ill e r
i
X .
x i ph i d i u m
X .
ande r s i
X .
[Figure]
a c u l a t u s _ J p W il d
[Figure]
s j one
[Figure]
ii
[Figure]
ape ll
[Figure]
[Table]
5. L INK P REDICTION 5.3. R ESULTS ON THE R EAL -W ORLD T AXONOMIES
In this section
[Table]
we evaluate the performances on the link prediction task for real-world taxonomies
[Table]
Table 2 presents the results in 32 -d embedding spaces for baselines and 16 -d complex hyperbolic space for UnitBall.
Predicting missing links requires generalization capacity, and UnitBall still has the best performances on all three datasets.
Besides, we see that Euclidean shows shortages on these hierarchically-structured data, which is consistent with the results in previous works (Nickel and
Kiela, 2017; 2018).
Similar to the results on the graph reconstruction task, Poincar´e and Hyperboloid have very close performances, while Hyperboloid has slightly better results.
They have signiﬁcant improvements over Euclidean, but they still fall behind UnitBall, which demonstrates our claims that the non-constant negative curvature of the complex hyperbolic space addresses the varying hierarchical structures on real-world taxonomies.
We notice that TreeRep does not perform well on the link prediction task.
As mentioned in Section 2, the combinatorial construction-based embedding methods target minimizing the reconstruction distortion of data.
However, minimizing the reconstruction distortion may overﬁt the training set, thus resulting in the unpromising generalization performance for unobserved edges.
Hence, they are more suitable to learn the representation of graph data without missing links.
The evaluation of
TreeRep on the graph reconstruction task can be referred to Section 5.2.1, Appendix F.5 and F.6.
5.3. E XPLORING THE E MBEDDING D IMENSIONS
In this section, we explore the performances in different embedding dimensions.
The results on
YAGO3-wikiObjects are presented in Table 3.
Results on other datasets are in Appendix F.7.
We ﬁnd that with the increase of the embedding dimension, Euclidean can have big improvements, but its performances in 128 -d still cannot surpass other methods in 8 -d.
TreeRep also achieves better results with the increase of dimension, but overall its performances on the link prediction task are not very promising.
By comparison, Poincar´e, Hyperboloid, and UnitBall achieve great results steadily.
8 -d is already enough for Poincar ´ e and Hyperboloid to handle the link prediction task while
[Table] [Table]
UnitBall has small improvements from 4 -d to 16 -d, then converges to the stable performance.
The results demonstrate that the Euclidean embeddings need to increase the dimension to better model the hierarchies, while the complex hyperbolic space and the hyperbolic space have strong generalization competence for hierarchical structures.
5.3. E XPLORING 1 - N S TRUCTURE
A noteworthy difference between the real-world taxonomies and the tree structures is that the taxonomies contain many 1 - N ( 1 child links to multiple parents) cases while in a tree each node except the root is linked to only 1 parent node.
To investigate the advantages of complex hyperbolic geometry on the 1 - N structures, we evaluate the performances of UnitBall, Poincar´e, and Hyperboloid on predicting the 1 - N edges.
We evaluate on YAGO3-wikiObjects since it contains abundant 1 - N structures.
The statistics of the 1 - N structures in YAGO3-wikiObjects original taxonomy ( original means without computing the transitive closure) are given in Figure 2.
In this experiment, we train the embeddings on the full transitive closure of YAGO3-wikiObjects and then predict the 1 - N edges.
The Hits@10 scores are reported on Table 4.
For results of other metrics, please refer to Appendix F.8.
We can see that UnitBall has a very small compromise for the 1 - 1 edges, i.e., the edge pattern of tree structures.
Furthermore, UnitBall outperforms the hyperbolic models largely on 1 - N structures for N > 1 .
Even for nodes that link to more than 20 parents, UnitBall can have accurate top 10 predictions for these edges.
The results demonstrate that the complex hyperbolic embeddings maintain the advantages in the edge pattern of tree structures as well as handling more complicated hierarchical structures compared with the real hyperbolic embeddings.
C ONCLUSION AND F UTURE W ORK
In this paper, we present a novel approach for learning the embeddings of hierarchical structures in the unit ball model of the complex hyperbolic space.
We characterize the geometrical properties of the complex hyperbolic space and formulate the embedding algorithm in the unit ball model.
We exemplify the superiority of our approach over the graph reconstruction task and the link prediction task on both synthetic and real-world data, which cover the various hierarchical structures and two speciﬁc structures, namely multitree structure and 1 - N structure.
The empirical results show that our approach outperforms the hyperbolic embedding methods in terms of representation capacity and generalization performance.
Motivated by our theoretical grounding and empirical success, we believe the complex hyperbolic embeddings will have promising improvements on the knowledge graph embeddings, neural networks, and other related applications.
R EPRODUCIBILITY S TATEMENT
• Please see Section 3 and 4 for the full set of assumptions of all theoretical results.
Please see Appendix B and C for complete proofs of all theoretical results.
• Please see Section 5.1 and Appendix F.2, F.3 for the experimental details (e.g., data splits, hyperparameters, hardware, etc).
• The data used in the paper are publicly available.
We cite the corresponding references and give the public data links in Appendix F.1.
• The code is proprietary for this moment.
The code will be released after the the publication of the paper.
A M ORE P RELIMINARIES A. D EFINITION OF C URVATURE
The curvature describes the curve of Riemannian manifolds and controls the rate of geodesic deviation.
In this paper, curvature refers to the sectional curvature .
Deﬁnition 1 (Curvature) .
Given a Riemannian manifold and two linearly independent tangent vectors at the same point, u and v , the (sectional) curvature is deﬁned as
[Equation]
where R is the Riemann curvature tensor, deﬁned by the convention R ( u , v ) w = ∇ u ∇ v w − ∇ v ∇ u w − ∇ [ u , v ] w .
Here ∇ indicates the Levi-Civita connection, whose deﬁnitions are given below.
Before deﬁning Levi-Civita connection, we need to ﬁrst deﬁne the afﬁne connection.
Deﬁnition 2 (Afﬁne connection) .
Let M be a smooth manifold and let Γ( T M ) be the space of vector ﬁelds on M , i.e., the space of smooth sections of the tangent bundle T M .
Then an afﬁne connection on M is a bilinear map
[Equation]
such that for all f in the set of smooth functions on M , written C ∞ ( M, R ) , and all vector ﬁelds
X, Y on M :
1. ∇ fX Y =
f ∇ X Y , i.e., ∇ is C ∞ ( M, R ) -linear in the ﬁrst variable; 2.
∇ X ( fY ) =
∂
X fY + f ∇ X Y , where ∂ X denotes the directional derivative, i.e., ∇ satisﬁes Leibniz rule in the second variable.
Next, we deﬁne the Levi-Civita connection.
Deﬁnition 3 (Levi-Civita connection) .
An afﬁne connection ∇ is called a Levi-Civita connection if
it preserves the metric, i.e., ∇ g = 0 .
it is torsion-free,
i
e., for any vector ﬁelds X and Y we have ∇ X Y − ∇ Y X =
[ X, Y ] , where [ X, Y ] is the Lie bracket of the vector ﬁelds X and Y .
A. T HE D ISTANCE F UNCTION ON THE U NIT B ALL M ODEL
The distance function in Eq.
(9) maintains the tree-like metric properties.
When the points are very close to the origin, it approximates to the Euclidean distance.
Additionally, when a point is closer to the origin, it has relatively smaller distances to the other points.
Correspondingly, the points near the boundary have very large distances from each other.
Therefore, in ideal conditions, the root node of a tree is embedded in the origin while the deeper nodes are embedded farther away from the origin.
Recall that the distance function in the real hyperbolic space (Nickel and Kiela, 2017) has similar properties since the real hyperbolic space, as the totally geodesic subspace of the complex hyperbolic space, inherits the tree-like metrics.
More details about the Bergman metric and distance function can be referred to Chapter 3.1 in (Goldman, 1999).
B P ROOF OF T HEOREM
In Section 3.2 in the paper, we presented Theorem 1 about the curvature of the complex hyperbolic space H n C (Goldman, 1999).
The sketch explanation is that all unit tangent vectors are equivalent, but not all directions are spanned by two unit tangent vectors.
Before proving Theorem 1, we need to introduce the deﬁnition of K¨ahler structure (Mok, 1989).
Deﬁnition 4 (K¨ahler structure) .
A K¨ahler structure can be deﬁned in any of the following equivalent ways:
A complex structure with a closed, positive (1 , 1) -form.
A Riemannian structure with a complex structure such that the corresponding exterior 2 -form is closed.
A symplectic structure with a compatible integrable almost complex structure which is positive.
Recall that in Section 3.2, we deﬁned the complex hyperbolic space H n C using the projectivization of the negative zone with a Hermitian form (cid:104)(cid:104) z , w (cid:105)(cid:105) .
Denote ω as the imaginary part of the Hermitian form (cid:104)(cid:104) , (cid:105)(cid:105) , i.e., ω ( z , w ) =
12 i ( (cid:104)(cid:104) z , w (cid:105)(cid:105) − (cid:104)(cid:104) w , z (cid:105)(cid:105) ) , then according to (Goldman, 1999), the metric ω is positive and closed , and necesssarily has type (1 , 1) .
Then by the ﬁrst deﬁnition in Deﬁnition 4, H n
C is a K¨ahler structure.
Let M be a K¨ahler manifold and z ∈ M .
Denote T z M as the tangent space of M at z and
J : T M → T M is an endomorphism.
As proved in (Kobayashi and Nomizu, 1963), the curvature of real 2 -planes in the tangent space
T z M has the following properties:
Theorem 2.
Let M be a connected K¨ahler manifold of complex dimension n ≥ 2 .
If the holomorphic sectional curvature K ( p ) , where p is a plane in T z M invariant by J , depends only on z , then M is a space of constant holomorphic sectional curvature.
Next, we give a proposition in (Kobayashi and Nomizu, 1963), which is about the curvature of a plane.
Proposition 1.
If u , v is an orthonormal basis for a plane p and if we set the curvature of p as
K ( p ) =
R ( u , v ) , where R ( u , v ) is the Riemann curvature tensor, then
[Equation]
where α ( p ) is the angle between p and J ( p ) .
Finally, we prove Theorem 1 as follows.
Proof.
Let M be a K¨ahler manifold and z ∈ M .
From Theorem 2, the corresponding sectional curvature function of real 2 -planes in T z M is completely determined by the sectional curvature function restricted to complex lines in T z M .
If the sectional curvature of every complex line in T M equals κ , then M is said to have constant holomorphic sectional curvature κ .
Then from Proposition 1, we can know that in this case, the sectional curvature of a 2 -dimensional subspace S ⊂ T M is
[Equation]
where α ( S ) is the angle of holomorphy, deﬁned as the smallest angle between two nonzero vectors from two linear subspaces of the underlying real vector space of M .
In particular, the complex hyperbolic space H n C is a K¨ahler structure with κ = − 1 .
Since 0 ≤ cos 2 α ( S ) ≤ 1 , then from Eq.
(17), we can have − 1 ≤ K ( S ) ≤ − 1 / 4 for any 2 -dimensional subspace S ⊂ T M of H n C , i.e., the (sectional) curvature is not constant in H n C , but pinched between − 1 and − 1 / 4 .
Thus we proved the non-constant curvature of H n C .
Speciﬁcally, we discuss the complex projective lines and totally real planes in the unit ball model of the complex hyperbolic space:
[Equation]
First let’s consider the case of complex projective lines.
Consider a complex line L in C n that intersects the unit ball model B n C .
Let z be any point in L ∩ B n C .
We can apply an element of PU ( n, 1) to L so that it becomes the last coordinate axis { ( 0 , z n ) | z n ∈ C } , whose intersection with B n C is the disk | z n | < 1 .
Then the restriction of the Bergman metric to this disc is the Poincar´e metric (Beardon, 2012) of constant curvature − 1 .
In order to see this, let z = ( 0 , z n , 1) and w = ( 0 , w n , 1) , z , w ∈ L ∩ B n C , then from Eq.
(9) in
Section 4.1, the distance between z and w is given by Eq.
Then we have
[Equation]
which is just the Poincar´e metric (Beardon, 2012).
Next consider a totally real plane p .
Any totally real plane p is the image under an element of
PU ( n, 1) of the subspace comprising those points of B n C with real coordinates, that is actually an embedded copy of the real hyperbolic space H n R = { ( x 1 , . . .
, x n ) | x 1 , . .
, x n ∈ R } .
This subspace intersects B n C in the subset consisting of those points with x 21 + · · · + x 2 n < 1 .
Then the Bergman metric restricted to this real-space unit ball is just the Klein-Beltrami metric (Ratcliffe et al., 1994) on the unit ball in R n with constant curvature − 1 / 4 .
To see this, let x =
x 1 , . .
x n , 1) and y =
( y 1 , . . .
, y n , 1) , x , y ∈ H n R ∩ B n C , then apply the similar process with the above, we have
[Equation]
which is the Klein-Beltrami metric (Ratcliffe et al., 1994) on the unit ball in R n with constant curvature − 1 / 4 .
Therefore, we proved that the curvature of H n C is − 1 in the directions of complex projective lines while − 1 / 4 in the directions of totally real planes.
Remark.
Curvature in the complex hyperbolic space is a very complicated topic in geometric group theory and differential geometry.
The complex projective lines and the totally real planes are two kinds of special subspaces, whose curvatures are presented above.
For the subspace that lives in between the two special cases, its curvature can be computed accordingly.
Since digging into all the curvature details of the complex hyperbolic geometry is not the essential part of our work, we have omitted the related content in our paper.
We refer the interested readers to (Fisher) for an interesting example in the complex hyperbolic space (its curvature differs with our work with a constant multiplier 4 ).
In Section 5 and Figure 5 of (Fisher), the author explored the curvature of a triangle in complex hyperbolic geometry with numerical computation.
C D ERIVATION OF D ISTANCE G RADIENT IN THE U NIT B ALL M ODEL
The distance function in the unit ball model is given by Eq.
We need to compute the distance gradient ∇
[Equation]
during the Riemannian optimization.
The full derivation is as follows.
First, we need to introduce Wirtinger derivatives (Wirtinger, 1927), which constructs a differential calculus for differential functions on complex domains.
Deﬁnition 5 (Wirtinger derivatives) .
The partial derivatives of a (complex) function f ( z ) of a complex variable z
= x + iy ∈ C , x, y ∈ R , with respect to z and ¯ z = x − iy , respectively, are deﬁned as:
[Equation]
The Wirtinger derivatives can be rewritten as:
[Equation]
Let p = cosh( d B n C ( z , w )
[Equation]
= 2 (cid:104)(cid:104)
[Equation]
, w (cid:105)(cid:105)(cid:104)(cid:104) w
[Equation]
z
[Equation]
(cid:104)(cid:104) z , z (cid:105)(cid:105)(cid:104)(cid:104) w , w (cid:105)(cid:105) − 1 , then d B n C ( z , w ) =
arcosh ( p ) =
ln( p + (cid:112) p 2 − 1) .
Let z =
( z 1 , . . .
, z n , 1) ∈ B n C , then
[Equation] [Equation]
for 1 ≤ j ≤ n .
Similarly, we can have
[Equation] [Equation]
Then by Eqs.
[Equation]
and (24), we obtain ∂d B n C
[Equation]
Similarly, by Eqs.
(22), (23), and (24), we can get ∂d B n C ∂y j =
[Equation]
(cid:104)(cid:104)
[Equation]
(cid:105)(cid:105)(cid:104)(cid:104)
w
[Equation]
(cid:105)(cid:105)
[Equation]
(cid:104)(cid:104)
z
[Equation]
z (cid:105)(cid:105) 2 (cid:104)(cid:104) w
[Equation]
w (cid:105)(cid:105) (cid:17) , (26) where Re ( · ) and Im ( · ) denote the real and the imaginary part respectively.
Then we can have ∂d B n C
[Equation]
which are Eqs.
(13) and (14) in Section 4.3.
D D EFINITION OF δ -H YPERBOLICITY
In this section, we give the deﬁnition of δ -hyperbolicity (Gromov, 1987), which measures the treelikeness of graphs.
The lower δ corresponds to the more tree-like graph.
Trees have 0 δ -hyperbolicity.
Deﬁnition 6 ( δ -hyperbolicity) .
Let a, b, c, d be vertices of the graph G .
Let S 1 , S 2 and S 3 be
S 1 = dist ( a, b ) + dist ( d, c ) , S 2 = dist ( a, c ) + dist ( b, d ) , S 3 = dist ( a, d ) + dist ( b, c ) .
Suppose M 1 and M 2 are the two largest values among S 1 , S 2 , S 3 and M 1 ≥ M 2 .
Deﬁne hyp ( a, b, c, d ) =
M 1 − M 2 .
Then the δ -hyperbolicity of G is deﬁned as
[Equation]
, c, d ) .
That is, δ ( G ) is the maximum of hyp over all possible 4 -tuples ( a, b, c, d ) divided by 2 .
E M ULTITREE S TRUCTURE
In combinatorics and order-theoretic mathematics, a multitree structure is a directed acyclic graph (DAG) in which the set of vertices reachable from any vertex induces a tree, or a partially ordered set (poset) that does not have four items a , b , c , and d forming a diamond suborder with a ≤ b ≤ d and
[Figure]
XcorteziXnezahuacoyotlXmontezumaeXnigrensisXmultilineatusXpygmaeusXcontinensXmayaeXalvareziXhelleriiXsignumXmonticolusXclemenciae
[Figure]
XmontezumaeXnigrensisXmultilineatusXpygmaeusXcontinensXmeyeriXgordoniXcouchianusXxiphidiumXvariatusXandersiXmilleriXevelynaeXmaculatus
[Figure]
XcorteziXnezahuacoyotlXmontezumaeXnigrensisXmultilineatusXpygmaeusXcontinensXmayaeXalvareziXsignumXhelleriiPsjonesiiPriapella
[Figure]
[Figure]
a ≤ c ≤ d but with b and c incomparable to each other (also called a diamond-free poset (Griggs et al., 2012)).
Obviously, the multitree structure is not a tree since one child node can have multiple parents in multitree.
Note that the multitree structure is also different with 1 - N structure since the multitree has more strict conditions, that is, the multitree is a diamond-free poset.
By comparison, the 1 - N structure is more general in taxonomies.
For example, the subgraph of YAGO3-wikiObjects { ( Nei Gaiman , is-a , British screenwriters ), ( Neil Gaiman , is-a , British fantasy writers ), ( British screenwriters , is-a ,
British writers ), ( British fantasy writers , is-a , British writers ) } is a 1 - N structure, but it is not a multitree structure, because British screenwriters and British fantasy writers are incomparable to each other, i.e., there is no partial order between them.
Recall that our synthetic compressed graph is also aggregated from multiple trees, but it is not the multitree either.
A compressed graph is aggregated from multiple random trees on the same set of nodes while the trees in a multitree structure share the same leaf nodes.
Multitrees are widely used to represent multiple overlapping taxonomies over the same ground set.
The dataset Xiphophorus is a multitree structure containing 160 trees on 26 leaf nodes.
Some examples of its subtrees are in Figure 3 and its cloud tree plot is in Figure 2.
F M ORE E XPERIMENTS F. D ATA
In Section 5.1.1 in the paper, we introduced how we generate the synthetic data:
[Table]
; manifold: unitball ; manifold: unitball ; manifold: unitball ; learningrate: 8 ; learningrate: 11 ; learningrate: 14 ; learningrate: 12 ; epochs: 1500 ; epochs: 200 ; epochs: 1200 ; epochs: 900 ; dampening: 0 . 75 ; dampening: 1 . 0 ; dampening: 1 . 0 ; dampening: 1 . 0 ; burnin: 20 ; burnin: 20 ; burnin: 20 ; burnin: 20 ; burninmultiplier: 0 . 01 ; burninmultiplier: 0 . 01 ; burninmultiplier: 0 . 01 ; burninmultiplier: 0 . 01 ; negativesample: 50 ; negativesample: 50 ; negativesample:
[Table]
; negativesample: 50 ; negativemultiplier:
[Table]
; negativemultiplier:
[Table]
; negativemultiplier:
[Table]
; negativemultiplier
[Table]
; maxnorm
[Table]
maxnorm
[Table]
maxnorm
[Table]
maxnorm
[Table]
Synthetic.
We generate various balanced trees and compressed graphs using the NetworkX package (Hagberg et al., 2008).
For balanced trees , we generate the balanced tree with degree r and depth h .
For compressed graphs , we generate k random trees on m nodes and then aggregate their edges to form a graph.
We give some examples of the synthetic data in Figure 4.
As we can see, the compressed graphs- ( m = 7 , k = 1 ) are random trees on 7 nodes, so their δ -hyperbolicities are 0 .
The compressed graph-( m = 7 , k = 2 ) is no longer a tree after aggregating from two trees.
Its local structures are more varying and complicated.
Here we also give the public links of the real-world data used in our experiments:
Xiphophorus (Cui et al., 2013), 6 ICD10 (Br¨amer, 1988), 7 YAGO3-wikiObjects (Mahdisoltani et al., 2015), 8 WordNetnoun (Miller, 1995).
F. H ARDWARE
We conduct all the experiments except TreeRep on four NVIDIA GTX 1080Ti GPUs with 8 GB memory each.
For TreeRep, we need more memory to store the distance matrices, so we use a 96 -core
NVIDIA T
GPU server with 503 GB memory.
F. H YPERPARAMETERS
For the baselines (TreeRep (Sonthalia and Gilbert, 2020), 10 Euclidean, Poincar´e (Nickel and Kiela, 2017), and Hyperboloid (Nickel and Kiela, 2018)), 11 we use their public codes to train the embeddings.
For all methods, we tune the hyperparameters by grid search.
For the graph reconstruction task, we tune the hyperparameters on balanced tree-(15,3) in 20 -dimensional embedding spaces ( 10dimensional complex hyperbolic space for UnitBall), while for the link prediction task, we tune the hyperparameters on the validation sets in 32 -dimensional
embedding spaces ( 16 -dimensional complex hyperbolic space for UnitBall).
The hyperparameters are given in Table 5.
F. E VALUATION
Our evaluation closely follows the setting of (Nickel and Kiela, 2017; 2018), which infers the hierarchies from distances in the embedding space.
Speciﬁcally, for each test edge ( z, w ) , we compute the distance between the embeddings d B n C ( z , w ) and rank it among the distances of all unobserved edges for z : { d B n C ( z , w (cid:48) ) :
( z, w (cid:48) ) / ∈ Training } .
We then report the following evaluation metrics of the rankings.
Denote E test as the test edge set and V = { z |∃ w, ( z,
w ) ∈
E test } as the test node set.
Let NE z = { w 1 , w 2 , . .
. , w | NE z | } be the ground truth neighbor set of node z .
Mean average precision (MAP).
The average precision (AP) is a way to summarize the precisionrecall curve into a single value representing the average of all precisions and the MAP score is calculated by taking the mean AP over all classes.
For a node z , from the learned embeddings, we can obtain the nodes closest to its embedding z .
Let R z,w i be the smallest set of such nodes that contains w i (the i -th neighbor of z ).
Then the MAP is deﬁned as:
[Equation]
Mean reciprocal rank (MRR).
The MRR is a statistic measure for evaluating a list of possible responses to a sample of queries, ordered by the probability of correctness.
For a node z , from the learned embeddings, we can rank its distances with other nodes from the smallest to the largest.
Let rank w i be the rank of w i (the i -th neighbor of z ).
Then the MRR is deﬁned as:
[Equation]
The proportion of correct types that rank no larger than N (Hits@ N ).
Hits@
N measures whether the top N predictions contain the ground truth labels.
For a node z , from the learned embeddings, we can obtain the set of N nodes closest to its embedding z , denoted as R Nz .
Then
the
Hits@
N is deﬁned as:
[Equation]
where I ( | R Nz ∩ NE z | ≥ 1) is the indicator function.
F. G RAPH R ECONSTRUCTION R ESULTS ON B ALANCED T REES
To compare the representation capacities of UnitBall and the hyperbolic embedding models for the tree structures, we evaluate the graph reconstruction task on the synthetic balanced trees.
A balanced tree- ( r, h ) has degree r and depth h , so it has r 0 + · · · + r d nodes and r 0 + · · · + r d − 1 edges.
The δ -hyperbolicity of any balanced tree is 0 .
We embed the balanced trees into 20 -d hyperbolic space for the baselines and 10 -d complex hyperbolic space for UnitBall.
Figure 5 presents the MAP and Hits@3 scores with varying r and h .
We see that when the tree is in small scale, e.g., ( r, h ) =
(15 , 3) , (10 , 2) , (10 , 3) , all methods have very good performances,
[Figure]
[Table]
demonstrating the expected powerful capacities of hyperbolic geometry and complex hyperbolic geometry on tree structures.
However, when the breadth or the depth increases, the performances of
Poincar´e and Hyperboloid drop rapidly, suggesting that the optimization-based embeddings in H 20 R are not effective enough for reconstructing trees of such scales.
In comparison, UnitBall and TreeRep achieve stable performances for larger trees.
TreeRep learns a tree structure from the data as an intermediate step and then embeds the learned trees into the hyperbolic space using Sarkar’s construction (Sarkar, 2011).
When the input data is a tree, TreeRep exactly recovers the original tree structure.
Figure 5 shows that UnitBall achieves comparable or even better performances than TreeRep on the balanced trees.
The results demonstrate that UnitBall does not compromise on trees.
It produces high-quality embeddings for tree structures.
F. C OMPARISON WITH T REE R EP ON R EAL -W ORLD T AXONOMY R ECONSTRUCTION
In this section, we compare UnitBall with TreeRep on the real-world taxonomy reconstruction task.
The results are presented in Table 6.
As we analyzed in Section 5.3.1, TreeRep, as a combinatorial construction-based embedding method, is more suitable for the graph reconstruction task.
Its performance is much better than that on the link prediction task.
In addition, UnitBall still outperforms
TreeRep on reconstructing real-world taxonomies.
We also notice the memory issues of the combinatorial construction-based embedding methods.
Although TreeRep is very efﬁcient in embedding tree structures since it does not need the gradientbased optimization steps, it costs more memory resources for constructing the tree structures from data.
It is basically a computation time vs. memory cost trade-off issue.
For a graph with m nodes,
TreeRep needs to construct a matrix of size c · m × c · m to construct the tree structure, where 1 ≤ c ≤ 2 is a hyperparameter.
We report the memory cost ( GiB ) in Table 6.
UnitBall costs much less memory to learn the embeddings.
F. M ORE R ESULTS ON V ARIOUS D IMENSIONS
In Section 5.3.2, we reported the performances in different embedding dimensions on YAGO3wikiObjects because of the page limits.
Here we present the results in 8 -d, 32 -d, and 128 -d embedding spaces ( 4 -d, 16 -d and 64 -d complex hyperbolic spaces for UnitBall) on ICD10 and WordNet-noun in
Table 7.
Again, we see that with the increase of the embedding dimension, Euclidean can have big
[Table] [Table]
improvements, but its performances in 128 -d still cannot surpass UnitBall and the hyperbolic models in 8 -d.
UnitBall outperforms the baselines almost all the time.
Although on WordNet-noun, UnitBall in 4 -d has slightly lower MAP and MRR than Poincar´e and Hyperboloid in 8 -d, it has much higher
Hits@3.
F. M ORE R ESULTS ON 1 - N S TRUCTURE
In Section 5.3.
[Table]
we explored the performances on 1 - N structures.
Here we present results of more evaluation metrics in Table 8.
The results of UnitBall are close with Poincar´e and Hyperboloid in 1 - 1 structure.
Nevertheless, UnitBall outperforms the hyperbolic baselines by a big margin in 1 - N structure for N > 1 .
Moreover, Unitball has considerably huge improvement for N > 6 , where the hyperbolic embedding models fail to make accurate predictions
[Table]
F.
C OMPARISON WITH T RAINABLE C URVATURE M ETHOD A TT H
Our work focuses on the representation of single-relation graphs, which is a different research topic with multi-relational graph embeddings or knowledge graph embeddings, so it is hard to ﬁnd an appropriate experimental setting to compare them.
Nevertheless, to address the concerns of comparison with the trainable curvature method, here we evaluate AttH (Chami et al., 2020) on the single-relation taxonomy link prediction task.
We tune the hyperparameters on the validation set and report the mean results over 5 running executions.
[Table] [Table]
From the results in Table 9, we see that UnitBall outperforms AttH in the single hypernymy relation link prediction task.
However, UnitBall cannot infer multiple relations like AttH for now.
MWe believe the future work of the complex hyperbolic embeddings will have promising improvements on multi-relational graph embeddings.
F. C OMPARISON WITH H YPERBOLIC GNN S
Our work focuses on the single-relation graph embeddings and taxonomy embeddings, so we do not evaluate the neural networks in our tasks in the main body of the paper.
Although hyperbolic
GNNs also involve graph embeddings and can deal with the link prediction task, they make use of not only the edges between nodes but also the node features.
The message propagation and attention mechanism make GNNs more ﬂexible to handle various downstream tasks than shallow embeddings.
In this section, we evaluate UnitBall on the link prediction task on the datasets of GIL (Zhu et al., 2020).
The results of HGCN (Chami et al., 2019) and GIL (Zhu et al., 2020) are copied from Table 2 of the GIL’s original paper (Zhu et al., 2020).
We strictly follow their experimental settings and report the mean results of UnitBall in ROC AUC over 5 running executions.
The dimension is 8 for
UnitBall and 16 for HGCN and GIL.
We can see that UnitBall outperforms HGCN on the ﬁve datasets.
GIL is slightly better than UnitBall on most datasets while being outperformed by UnitBall on Pubmed.
The results are very promising for UnitBall since UnitBall is a shallow embedding approach without deep architecture or feature interaction.
We believe the complex hyperbolic embeddings will help to improve the GNNs and bring more insights into geometric deep learning.
F. C OMPARISON WITH P RODUCT E MBEDDINGS
As mentioned in Section 2, the product space embeddings (Gu et al., 2019) tackles the challenges in varying local structures by jointly learning the curvature and the embeddings of data in a product manifold.
Although it is impractical to search for the best manifold combination among enormous combinations for each new structure, it is worth exploring the comparisons between the complex hyperbolic embeddings and products of hyperbolic embeddings.
Therefore, in this section, we conduct experiments on the reconstruction task in synthetic compressed graphs.
We evaluated the 16 -dimensional UnitBall complex hyperbolic embeddings and 32 -dimensional product hyperbolic embeddings on ( H 2 R ) 16 , ( H 4 R ) 8 , ( H 8 R ) 4 .
The results are reported in Table 11.
Recall that each compressed graph- ( m, k ) consists of m nodes and is aggregated from k random trees on the m nodes.
The bigger k corresponds to the denser and noisier graph.
When k = 1 ( δ = 0 ), the graph is exactly a tree structure, UnitBall and the product hyperbolic embeddings both have much better performances in this case.
When k > 1 , UnitBall still outperforms the product hyperbolic embeddings by a large margin.
Especially when m = 500 , k = 2 , 3 , the δ is big, which means the graph deviates from tree structures a lot, the product hyperbolic embeddings fail to reconstruct the graph while UnitBall successfully handles the noisy structures.
[Table]
Compressed graph- ( m, k ) (MAP) m (k=5)         1000 δ -hyperbolicity 1.
Product- ( H 2 R )30.
Product- ( H 4 R )32.
Product- ( H 8 R )31.
UnitBall- H 16 C 47.
Compressed graph- ( m, k ) (Hits@3)
Product- ( H 2 R )38.
Product- ( H 4 R )46.
Product- ( H 8 R )54.
UnitBall- H 16 C 84.
[Table]
F. C ASE S TUDY OF N ODES E MBEDDED IN THE T OTALLY R EAL P LANE
Although as analyzed in Appendix B, curvature in the complex hyperbolic space is a very complicated topic in geometric group theory and differential geometry.
Computing the curvature of an arbitrary point in the complex hyperbolic space or visualizing the curvatures everywhere are highly complicated problems, the existing work usually explored a few examples by numerical computation.
However, the curvatures in two special subspaces, i.e., the complex projective lines and the totally real planes, are well-studied.
It is interesting to see which nodes lie in the special subspaces.
In this section, we ﬁnd the nodes of the real-world taxonomies that are embedded in the most simple totally real plane of the unit ball model, i.e., H n R ∩ B n C , where H n R = { ( x 1 , . . .
, x n ) | x 1 , . .
, x n ∈ R } .
Table 12 shows that the root node of all three taxonomies are embedded in the totally real plane, which is in accordance with expectations since the root nodes are usually embedded close with the origin.
For the more ﬁne-grained taxonomy YAGO3-wikiObjects, some high-level nodes are also embedded in this totally real plane.
These nodes actually form a chain of hypernymy relation in the taxonomy: Men → Organisms → Physical objects → Objects .
Recall Theorem 1 states that the curvature in complex hyperbolic space is pinched between − 1 (in the directions of complex projective lines) and − 1 / 4 (in the directions of totally real planes).
From the results in the three taxonomies, we can see that the high-level nodes in the hierarchical structure tend to lie in the totally real planes, which are the least curved subspaces of the complex hyperbolic space.