Manifold Alignment across Geometric Spaces for Knowledge Base Representation Learning
Abstract
Knowledge bases have multi-relations with distinctive properties.
Most properties such as symmetry, inversion, and composition can be handled by the Euclidean embedding models.
Nevertheless, transitivity is a special property that cannot be modeled eﬃciently in the Euclidean space.
Instead, the hyperbolic space characterizes the transitivity naturally because of its tree-like properties.
However, the hyperbolic space reveals its weakness for other relations.
Therefore, building a representation learning framework for all relation properties is highly diﬃcult.
In this paper, we propose to learn the knowledge base embeddings in diﬀerent geometric spaces and apply manifold alignment to align the shared entities.
The aligned embeddings are evaluated on the out-of-taxonomy entity typing task, where we aim to predict the types of the entities from the knowledge graph.
Experimental results on two datasets based on YAGO3 demonstrate that our approach has signiﬁcantly good performances, especially in low dimensions and on small training rates.
1. Introduction
Representation learning plays an important role in the knowledge base (or in general multirelational database) inference as well as its downstream tasks [Nickel et al., 2016].
Relations in knowledge bases have distinctive properties, such as symmetry, inversion, and composition.
Transitivity is a typical property in knowledge bases.
Transitive relations such as IsA are commonly used in many popular knowledge bases with taxonomies, such as YAGO [Suchanek et al., 2007] and WordNet [Miller, 1995].
Transitive relation has been shown quite diﬀerent from other relations.
While symmetry, inversion and composition can be easily handled by the Euclidean space [Sun et al., 2019], the Euclidean embedding methods suﬀer from severe limitations for transitive relations and tree-like structures [Linial et al., 1995].
In contrast, the hyperbolic space is capable of embedding any ﬁnite tree while preserving the distances approximately [Gromov, 1987], so the transitivity is naturally characterized by the hyperbolic space [Nickel and Kiela, 2017].
However, the hyperbolic space cannot achieve promising results on capturing various relation patterns [Kolyvakis et al., 2019, Balazevic et al., 2019, Chami et al., 2020] due to the incompatibility between the geometry and general graph structures.
To learn the embeddings of a wide variety of structures, [Gu et al., 2019] proposed to learn graph embeddings in a product manifold combining several spaces.
Nevertheless, the model also focuses on single-relation graphs, making it inapplicable to knowledge bases.
Therefore, it is diﬃcult to ﬁnd a uniﬁed space to characterize all relation properties because both the Euclidean and the hyperbolic embeddings tackle some relation properties while having weaknesses on others. Hence, it is natural to think about an alternative [Table] guitar
The same instance [Table] [Figure] approach that uses diﬀerent geometric spaces to best characterize various relation properties, while aligning the shared entities.
In this paper [Figure] we make use of manifold learning to address the issue.
We propose to embed the knowledge base in diﬀerent spaces according to the relation properties and then align the embeddings based on the local structures as well as the correspondences by manifold alignment [Ma and Fu, 2011], where the intersection entities act as the correspondences across spaces.
Manifold is a topological space that is locally Euclidean, i [Figure] e., around every point, there is a neighborhood that is topologically the same as the open unit ball in
Euclidean space.
Although the hyperbolic geometry diﬀers a lot from Euclidean geometry, the hyperbolic manifold is locally Euclidean.
We use the projections between the hyperbolic and the Euclidean embeddings to align the underlying local structures of the knowledge base.
Through manifold alignment, we can recover the correspondences across the sub-graphs as well as preserving their local structures.
We evaluate the aligned embeddings through an out-of-taxonomy entity typing task: given a knowledge base containing the taxonomy and the knowledge graph with their intersection entities, as Figure 1 shows, the goal is to predict the types of the out-of-taxonomy entities from the knowledge graph.
The task is important in practice since some entities are missing in real-world taxonomies but occur in the knowledge graphs.
Manual labeling is unpractical because of the large number of new entities and the high labeling cost.
In the empirical evaluation, our approach achieves signiﬁcant improvements.
Notably, our approach has superior performances even when the training taxonomy is very small.
2. Related Work
Knowledge graph embeddings.
Traditional knowledge graph embedding models can be translation-based [Bordes et al., 2013] or bilinear models [Nickel et al., 2011, Yang et al., 2015].
Several extensions have been made in the real Euclidean space
[Lin et al., 2015, Ji et al., 2015, Sun et al., 2019] and the complex Euclidean space
[Trouillon et al., 2016, Zhang et al., 2019].
It is shown that several properties of relations such as symmetry, inversion, and composition can be well handled by the Euclidean space [Sun et al., 2019].
Hierarchy-aware knowledge graph embeddings.
Some works made special eﬀorts for transitive relations in the Euclidean space.
TransC
[Lv et al., 2018] encoded types as spheres and entities as vectors for modeling the hypernymy relation HAKE [Zhang et al., 2020] proposed to map entities into the Euclidean polar coordinate system.
JOIE [Hao et al., 2019] separated the knowledge base into the taxonomy and the remaining knowledge graph, which had the same setting as our work.
It then leveraged a non-linear transformation between two Euclidean embedding models.
However, the non-linear aﬃne transformation is not powerful enough to build correlations between two underlying structures.
Hyperbolic embeddings.
Hyperbolic embeddings have gained much attention in recent years.
[Nickel and Kiela, 2017] proposed to use the Poncar´e ball model of the hyperbolic space to learn the graph embeddings.
[Ganea et al., 2018, Nickel and Kiela, 2018, B´ecigneul and Ganea, 2019, Sala et al., 2018, Gu et al., 2019, Sonthalia and Gilbert, 2020] then further improved hyperbolic embeddings.
These methods perform well on data with hierarchical structures and single transitive relation, but they cannot predict the out-of-taxonomy entities, thus are not directly applicable to our entity typing task.
Motivated by the above works,
MurP [Balazevic et al., 2019], AttH
[Chami et al., 2020], and HyperKA [Sun et al., 2020] explored the multi-relational knowledge graph embeddings in the hyperbolic space.
However, most relations in knowledge graphs do not have transitivity, thus do not ﬁt the hyperbolic space.
Their experimental results only signiﬁcantly improved on transitive relations or the datasets with natural hierarchical structures.
Manifold alignment
Manifold alignment [Ham et al., 2003, Ma and Fu, 2011] is a class of algorithms aligning the local structures and transferring knowledge across data sets.
In this work, we use two-step alignment [Lafon et al., 2006, Wang and Mahadevan, 2008], which utilizes the Laplacian eigenmaps [Belkin and Niyogi, 2003].
Note that our work focuses on a diﬀerent research topic with the knowledge graph entity alignment [Hao et al., 2016, Sun et al., 2020] and the ontology matching [Alvarez-Melis et al., 2020] since entity alignment/matching aligns the entities that referring to the same thing but having diﬀerent names, while in our work, the correspondences between the knowledge graph and the taxonomy are already known and we make use of the correspondences to apply manifold alignment and then predict the new out-of-taxonomy entities’ types.
3. Problem Formulation
Given a knowledge base B containing the taxonomy T and the knowledge graph G , we aim to learn the alignment between the embeddings of T and G .
An example of B is shown in Figure 1.
The taxonomy T contains the type set C
= { Thing , organization , person , scientist , artist } , the entity set I T
= { John Lennon , George Harrison } , and the directed edge set E T
= { e i,j : i, j ∈ C ∪ I T } , where e i,j represents that i and j have the transitive relation, e.g., ( John
Lennon , InstanceOf , artist ) and ( artist , SubClassOf , person ).
Denote the entity set and the relation set of the knowledge graph G as I G and R G respectively, 1 then G is composed of the triplets F = { ( h, r, t ) : h, t ∈ I G , r ∈ R G } , e.g., ( Eric
Clapton , wasBornIn , Guildford ).
There are some intersection entities between T and G , which we denote as I cor : I cor =
I T ∩ I G .
In Figure 1, I cor = { George Harrison } .
To apply manifold alignment, ﬁrst, we need to obtain the pretrained embeddings of I T and I G , denoted as I T and I G .
I T ∈
H n , I G ∈ R m , where H n is the n -dimensional hyperbolic space, and R m is the m -dimensional Euclidean space.
Then manifold alignment is applied to project I T
and I G into a shared manifold.
After that, we have the embeddings of all entities of B in the same manifold.
Note that here all entities do not include types in C in the taxonomy.
In the out-of-taxonomy entity typing task, the goal is to predict the types of the new entities from I n =
I G − I T .
It can also be regarded as a taxonomy completion task.
4. Approach
The main framework of our approach involves three parts: pretraining, manifold alignment, and retraining the taxonomy with new entities.
We give the details in the following.
4.1 Pretraining of Taxonomy and Knowledge Graph Embeddings
In our work, we use the hyperboloid model of hyperbolic embeddings [Nickel and Kiela, 2018] to learn the taxonomy embeddings.
The background of the hyperbolic space and the hyperboloid model are in Appendix A. For the knowledge graph, we employ TransE proposed by [Bordes et al., 2013], which is a classical and eﬀective embedding algorithm.
4.1.1 Hyperbolic Embeddings in the Hyperboloid Model
Given the taxonomy T with type set C , entity set I T , and edge set E T , the objective is to ﬁnd the embeddings of the types and entities T =
{ T i } , T i ∈ H n , where H n is the n -dimensional hyperboloid model ( H n is one model of H n ).
The soft ranking loss is
[Equation]
where N ( x i )
= { x k : ( x i , x k ) / ∈ E T } ∪ { x i } is the set of negative examples for x i together with x i .
The hyperboloid model H n and its distance function d h are deﬁned in Appendix A.
The minimization of L T makes the connected entities and types closer than those with no observed edges.
Riemannian SGD (RSGD)
[Bonnabel, 2013] is applied to train the hyperbolic embeddings.
4.1.2 TransE for Knowledge Graph Embeddings
Given a triplet ( h, r, t ), by regarding d ( h + r , t ) as the energy of ( h, r, t ), where h , r , t are the corresponding embeddings and d is some dissimilarity measure (usually the L 1 or L norm), the margin-based ranking loss over the knowledge graph G with the triplet set F is
[Equation]
for some margin γ > 0.
[ x ] + = max(0 , x ) and N ( h, r, t ) =
{ ( h (cid:48) , r, t ) | h (cid:48) ∈
I G } ∪ { ( h, r, t (cid:48) )
| t (cid:48) ∈
I G } is the negative sample set.
The optimization of L G encourages positive triplets to satisfy h
+ r ≈ t and negative ones
to satisfy that h + r is far away from t .
4.2 Manifold Alignment
After pretraining, we conduct manifold alignment to learn the aligned embeddings of the taxonomy entities I T and knowledge graph entities I G in a shared manifold.
The goal of manifold alignment is to ﬁnd the projection functions such that the projections not only minimize the distance between the corresponding points but also preserve the local manifold structures of the original data.
In our framework, given two entity sets I T , I G , and their embeddings I T ∈
H
n , I G ∈ R m , we aim to learn the projections φ I T : H n → R d and φ I G : R m → R d .
The algorithm works as follows:
First, we construct the binary correspondence matrix W using the intersection set I cor :
[Equation]
Next, we construct the adjacency graphs from I T and I G .
Speciﬁcally, we compute the pointwise distances within the two sets and select the k nearest neighbors ( k -nn) to construct the adjacency matrices A T and A G , i.e., A T ij = 1 if and only if entity i (or j ) is among the k nearest neighbors of entity j (or i ), and the same with A G .
Then we construct the similarity matrices S T and S G using heat kernel:
[Equation]
where t > 0 is the parameter of heat kernel.
d h refers to the hyperboloid distance in
Appendix A, and d is the L 2 norm in the Euclidean space.
The manifold alignment loss is deﬁned as:
[Equation]
where 0 ≤ µ ≤ 1 is the weight to balance between preserving the original manifold structures and minimizing the corresponding entity distances.
The ﬁrst term multiplied by µ infer that if two entities both from I T or both from I G are similar, their projections in the latent space should be close with each other.
If two entities from I T and I G are the same entity, the last term minimizes the distance between their projections in the shared manifold.
In practice, an additional constraint needs to be added in Eq.
(5) to avoid the all-zero solution.
More details are given in Appendix B.1.
Then minimizing L M is equivalent to solving a generalized eigenvalue problem of the joint graph Laplacian L : L v = λD v .
Denote the joint matrix J =
[Equation]
, D is a diagonal matrix and
D ii = (cid:80)
j J ji = (cid:80)
j J ij (the proof of J ’s symmetry can be referred to Appendix B.2), then the Laplacian L = D − J is a symmetric and positive semideﬁnite matrix (see Appendix
B.3 for proof).
Let v 0 , . .
. , v d be the solutions ordered according to their eigenvalues, i.e.,
L v i = λ i D v i for 0 ≤ i ≤ d , and 0 = λ 0 ≤ λ 1 ≤ · · · ≤ λ d .
Then the trivial eigenvector vis discarded and the next d eigenvectors are used for the aligned embeddings (proved in
Appendix B.4).
That means the embedding vectors for I T and I G in the shared manifold are
[Equation]
Following the above procedure, we project both entity sets into a shared manifold.
The aligned embeddings are learned by preserving the original manifold structure and recovering the correspondences.
They provide correlations between I T and I G .
4.3 Linking New Entities and Retraining
Once we obtain the aligned embeddings φ I T ( I T ) , φ
I G
( I G ) ∈ R d , we can compute the pairwise distances of the entities in the shared manifold.
From the pairwise distances, we further connect I T and I n =
I G − I T by k -nn and thus have the new edges E n = { ( x i , x j ) : x i ∈
I T , x j ∈ I n } .
Next, we retrain the hyperbolic embeddings of the completed taxonomy with E T ∪ E n .
Note that E n is diﬀerent with E T since E T is the set of original taxonomy edges representing the transitive relation among types and entities while edges in E n reveal the similarities between entities.
To diﬀerentiate them, we regard the edges in E n as weighted undirected edges during training.
That is to say, we append ( x i , x j ) and ( x j , x i ) to the training set if ( x i , x j ) ∈ E n .
Moreover, we let the terms associating with E n in the loss (Eq.
(1)) multiply by a weight, which adjusts the weight of the newly-added undirected edges.
Again we use RSGD to update the hyperbolic embeddings.
Then we can predict the links between the new entities I n and the types C according to the hyperboloid distance.
5. Experiments
In this section, we evaluate the performance of our approach on the out-of-taxonomy entity typing task.
We report the main results here.
For more experiments, please see Appendix D.
5.1 Experimental Settings 5.1.1 Data
We construct our knowledge bases from YAGO3 [Mahdisoltani et al., 2015], a huge semantic knowledge base derived from Wikipedia, WordNet, and GeoNames.
Consistent with our framework, YAGO3 is divided into the taxonomy and the knowledge graph.
We provide the
[Table]
statistics of our datasets in Table 1.
We also give their Gromov’s δ -hyperbolicity
[Gromov, 1987], which measures the tree-likeness of graphs (refer to Appendix C for deﬁnition).
The lower δ corresponds to the more tree-like graph.
We sample our datasets as follows:
YAGOwordnet.
In YAGO3, the taxonomy is formed with yago-types and yago-taxonomy .
They include relations SubClassOf and type .
The types are from Wikipedia and WordNet.
We extract all WordNet types and sample around 9K entities from yago-types and yagotaxonomy , then trace back from the sampled entities until the root to construct a hierarchy.
As in previous works [Nickel and Kiela, 2017, 2018], we compute the transitive closure of the hierarchy to construct the YAGOwordnet taxonomy.
wikiObjects.
From yago-types and yago-taxonomy , we extract all Wikipedia types and their descendants (including types and entities) to obtain a forest, where each tree has a root in Wikipedia types.
We then sample a subtree from the root (cid:104) wikicat Objects (cid:105) and compute its transitive closure to construct the wikiObjects taxonomy.
YAGOfacts
In YAGO3, yago-facts is the core knowledge graph which contains all facts (triplets) of YAGO3 that hold among entities.
Diﬀerent with yago-types and yago-taxonomy , yago-facts does not contain the transitive relations or any type.
We select around 50K most frequent entities from yago-facts and extract all triplets whose head entity and tail entity both belong to the selected entities to form YAGOfacts.
For each taxonomy (YAGOwordnet/wikiObjects), we randomly split the entities into training entities (50%) and test entities (50%) with the constraint that the test entities must occur in YAGOfacts, otherwise, YAGOfacts cannot provide any reference about the test entities.
Then we discard all edges containing any test entity to obtain the training taxonomy.
The training taxonomy and YAGOfacts are used for pretraining and the embeddings of the training entities are used for manifold alignment with embeddings of YAGOfacts’ entities.
The evaluation task is to predict the types of the out-of-taxonomy test entities.
The training taxonomy corresponds to T in Section 3 while YAGOfacts corresponds to G .
5.1.2 Baselines
The following baselines are compared with our approach ( GeoAlign ): KG embedding models, including TransE [Bordes et al., 2013], ComplEx [Trouillon et al., 2016], RotatE
[Sun et al., 2019]; hierarchy-aware Euclidean methods, including TransC [Lv et al., 2018],
HAKE [Zhang et al., 2020], JOIE [Hao et al., 2019]; multi-relational hyperbolic models:
MurP [Balazevic et al., 2019], AttH
[Chami et al., 2020], HyperKA
[Sun et al., 2020].
5.1.3 Training and Evaluation
To apply the baselines to our task, they are trained on all triplets of YAGOfacts combined with the training taxonomy, where the taxonomy edges are labeled as (cid:104) isA (cid:105) .
Then the test triplets are { ( x i , (cid:104) isA (cid:105) , C j ) } where x i is a test entity and C j is x
i ’s ground-truth type.
For knowledge graph embedding models, we use the OpenKE repository [Han et al., 2018] to train them while for other baselines, we use their public codes.
For all methods, we tune the hyperparameters on the knowledge base combining YAGOwordnet and YAGOfacts by grid search according to MAP score.
The hyperparameters are given in Appendix D.1.
The code and data of our work will be publicly available after the publishing of the paper.
We use the mean average precision (MAP), mean reciprocal rank (MRR), and the proportion of correct types that rank no larger than N (Hits@N) as our evaluation metrics, which are widely used for evaluating ranking and link prediction.
The details of prediction steps and the evaluation metrics are given in Appendix D.2.
In our experiments, each running is executed 5 times and the mean values of results are reported.
5.2 Overall Results
Table 2 presents the results in 50-dimensional embedding spaces.
The results show that
GeoAlign has the best performance on wikiObjects while having a very close performance with MurP on YAGOwordnet.
In fact, wikiObjects is more challenging since its taxonomy is more massive (see Appendix D.3 for the case study), making it more diﬃcult to ﬁnd all correct types for the out-of-taxonomy entity.
From Table 2, we see that the traditional Euclidean models (TransE, ComplEx, and
RotatE) are not capable of inferring the transitive relation, which is consistent with previous works on hyperbolic embeddings [Nickel and Kiela, 2017, 2018].
The hierarchy-aware methods (TransC, HAKE, and JOIE) have better results than the traditional Euclidean embeddings, but overall they cannot achieve comparative performances with the hyperbolic models.
For the multi-relational hyperbolic models, MurP and AttH, which use diﬀerent relation parameterizations on the base of Poincar´e embeddings, reveal their strengths.
MurP, AttH, and GeoAlign all take advantage of the hyperbolic geometry for the taxonomy embeddings, thus having close results on the entity typing task.
However, in Section 5.4, we will show that
GeoAlign has signiﬁcant improvements over MurP and AttH when the training taxonomy is very small.
Furthermore, MurP and AttH can only do the inference for all relation properties in the hyperbolic space, which is not suitable for the non-transitive relation properties, while GeoAlign can take advantage of any base embedding models rather than TransE for pretraining the knowledge graph. As for HyperKA, which leverages hyperbolic GNN for embeddings, we tried our best to tune the model, but it still cannot achieve promising results.
We think the neural models may not ﬁt this task setting and our datasets. That also accounts for why AttH is not as good as MurP. Compared with MurP, AttH adds the attention mechanism and has more complicated parameterizations. It may hurt the model’s feasibility sometimes.
[Table]
Dimension
MRR
Hits
[Table]
MRR
Hits
[Table]
MRR
Hits
[Table]
MRR
Hits
[Table] TransE [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
ComplEx
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
RotatE
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] HAKE [Table] [Table]
[Table]
[Table]
[Table]
[Table] JOIE [Table] [Table]
[Table]
[Table]
[Table] [Table] [Table] MurP
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
AttH
[Table]
[Table]
[Table]
[Table] [Table] [Table] HyperKA
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
5.3 Exploring the Embedding Dimensions
In this section, we explore the performances in diﬀerent embedding dimensions.
The results are presented in Table 3.
For GeoAlign, JOIE, and HyperKA, the embedding dimensions for the knowledge graph and the taxonomy are both set as n for n ∈ { 5 , 10 , 20 , 100 } .
From
Table 3, we see that with the increase of the embedding dimension, most methods get better results.
The Euclidean models can have big improvements in higher dimensions, such as
RotatE from 20-d to 100-d, but their 100-d performances cannot surpass GeoAlign in 5-d.
We also notice that some methods suﬀer from overﬁtting in high dimensions, e.g., TransC,
HAKE, and HyperKA drop down when the dimension increases.
In contrast, GeoAlign,
MurP, and AttH achieve great results steadily.
On the one hand, the hyperbolic models require much lower dimensions.
On the other hand, 5-dimensional embeddings are already [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Training/
[Table]
edges
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
enough for GeoAlign to learn the manifold alignment between the local structures of the knowledge base.
5.4 Results on Small Training Rates
In Section 5.2, we see that the performances of MurP, AttH, and GeoAlign are very close under the training rate=0 .
Here we explore their performances on smaller training rates.
For the training rate r ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } , we randomly split the entities into training entities ( r ) and test entities (1 − r ).
The splitting and training taxonomy construction procedure are the same as described in Section 5.1.1.
Note that the small training rate means that the pretrained taxonomy and the number of taxonomy entities used for manifold alignment are both small.
We report the MAP(%) scores in Table 4.
We ﬁnd that when the training rate is small, GeoAlign outperforms MurP and AttH signiﬁcantly.
With the increase of the training rates, their performances get more and more similar and converge to stable.
The results demonstrate the eﬀectiveness of our approach on small training rates.
5.5 Ablation Study 5.5.1 On the Retraining Step
To analyze the beneﬁts and the potential defects of the retraining step after manifold alignment, we compare the performances of GeoAlign with and without retraining on two tasks.
The ﬁrst task is the out-of-taxonomy entity typing task, which is the same as the above experiments.
The -w/o retraining method works in the way that after manifold alignment, we directly predict the types of the test entities as the types of its nearest neighbor according to the aligned embeddings.
If its nearest neighbor is not in the taxonomy’s training entities, we ﬁnd the next nearest one until it does.
The second task is graph reconstruction of the training taxonomy.
We intend to see whether the retrained model with new entities compromises the representation of the original taxonomy.
We report the results in Table 5.
The entity typing results demonstrate that after manifold alignment, the retraining of the embeddings with the added edges incorporates more structural information of the taxonomy with new entities and improves the performances signiﬁcantly.
The reconstruction results show that the retraining has little impact on the original training taxonomy.
The compromise is acceptable, especially when considering the remarkable improvements on the entity typing task (more than 10% MAP improvements in Table 5). [Table] [Table]
5.5.2 On the Hyperbolic Models
To compare the diﬀerent hyperbolic models
[Table]
we evaluate the performances of GeoAlign with the hyperboloid model and the Poincar´e ball model on two tasks.
Again, the ﬁrst task is the out-of-taxonomy entity typing task and the second task is graph reconstruction of the training taxonomy.
The MAP(%) and MRR(%) scores are reported in Table 6.
GeoAlignhyperboloid (Poincar´e) means we pretrain and retrain the taxonomy by the hyperboloid embeddings (Poincar´e ball embeddings).
From Table 6, we see that GeoAlign-hyperboloid surpasses GeoAlign-Poincar´e a lot, especially on the out-of-taxonomy entity typing task.
6. Conclusion and Future Work
We propose to learn the embeddings of knowledge bases in diﬀerent spaces and apply manifold alignment across the geometric spaces to build the projection.
The main motivation is to allow diﬀerent geometric spaces to model the various properties of relations as well as the various local structures of the knowledge base, while manifold alignment provides a way to incorporate the local manifold structure of two entity sets.
We propose a solid framework and evaluate our approach on an out-of-taxonomy entity typing task.
The empirical results demonstrate the superiority of our approach, especially in low dimensions and on small training rates.
Future works include the exploration of broader types of geometries for learning embeddings and more eﬀective approaches for aligning multiple manifolds. Acknowledgment
The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20) and the GRF (16211520) from RGC of Hong
Kong, the MHKJFS (MHP/001/19) from ITC of Hong Kong.
A. Hyperbolic Space Background
Hyperbolic space is a homogeneous space with constant negative curvature.
To describe hyperbolic space in mathematical language, there are ﬁve common-used models of hyperbolic space [Cannon et al., 1997], whose notations and deﬁnitions are as follows.
• The Half-space model
A n = { (1 , x 2 , . . .
, x n +1 ) :
x n +1 > 0 } .
• The Poincar´e ball model P n = { ( x 1 , . . .
x n , 0) : x 21 + · · · + x 2 n < 1 } .
• The Jemisphere model (or the Hemisphere model)
J n = { ( x 1 , . .
, x n +1 ) :
x 21 + · · · + x 2 n +1 = 1 , x n +1 > 0 } .
• The Klein model K n = { ( x 1 , . . .
x n , 1) : x 21 + · · · + x 2 n < 1 } .
• The hyperboloid model (or the Lorentz model)
H n = { ( x 1 , . .
, x n , x n +1 ) : x 21 + · · · + x 2 n − x 2 n +1 = − 1 , x n +1 > 0 } .
The ﬁve models of n-dimensional hyperbolic space H n are located in the ambient Euclidean space R n +1 and are isometric, i.e., for any two models ( X , d ) and ( X (cid:48) , d (cid:48) ), there exists a one-toone mapping f from X on to X (cid:48) preserving all distances: ∀ x, y ∈ X , d ( x, y )
=
d (cid:48) ( f ( x ) , f ( y )), and f is called isometry.
For representation learning in the hyperbolic space, the Poincar´e ball model [Nickel and
Kiela, 2017] and the hyperboloid model [Nickel and Kiela, 2018] are the two most popular models due to their intuitive formulas and nice properties.
Poincar´e ball model [Nickel and Kiela, 2017].
Denote the Riemannian manifold of
Poincar´e ball model as ( P n , ρ p ),
where P n = { x ∈ R n : (cid:107) x (cid:107) < 1 } , which represents the open n -dimensional unit ball in Euclidean space ( (cid:107) · (cid:107) is the Euclidean norm).
ρ p is the metric tensor, given in Table 7.
Hyperboloid model [Nickel and Kiela, 2018].
For the hyperboloid model ( H n , ρ h ), write H n using Lorentzian inner product: H n = { x ∈ R n +1 : (cid:104) x , x
(cid:105) L = − 1 , x 0 > 0 } , where (cid:104) x , y (cid:105) L = − x 0 y
+ (cid:80) ni
=1 x i
y
i is the Lorentzian inner product.
The manifold is composed of points on the forward sheet of a two-sheeted hyperboloid.
As mentioned before, Poincar´e ball model and the hyperboloid model are isometric.
The one-to-one mapping f : H n → P n is the central projection from the point (0 , . .
. , 0 , − 1):
[Equation]
Both Poincar´e ball model and the hyperboloid model are conformal models, i.e., they preserve angles of Euclidean space.
Their metric tensors and distance functions are given in Table 7.
The Poincar´e distance has the property that when closer to the origin, it approximates to the Euclidean distance.
Additionally, closer a point to the origin, the relatively smaller distances to other points it has.
Correspondingly, the points near the boundary have very large distances with each other.
Consider the tree structure, where the root node has relatively small distance with other nodes while leaf nodes are usually far away from each other.
Moreover, the volume of a ball in the Poincar´e model grows
[Table]
[Equation]
[Table]
[Equation]
[Table]
exponentially with its radius, resembling the tree-like property that the number of nodes grows exponentially with depth in a tree.
[Krioukov et al., 2010] built the approximate equivalence between hierarchical networks and the hyperbolic space.
Nevertheless, when using the Poincar´e ball model for hierarchical structures [Nickel and Kiela, 2017], numerical instabilities arise, which motivates the use of the hyperboloid model [Nickel and Kiela, 2018], since it can avoid the numerical instabilities from the fraction in distance function, hence allows for more eﬃcient computation on the manifold.
We provide the empirical evaluation of the two hyperbolic models in Section 5.5.2.
B. Proofs of Section 4.2 about Manifold Alignment B.1 The Loss Function of Manifold Alignment
As presented in Section 4.2, the loss function includes two parts, one for preserving the local manifold within each dataset, another for recovering the correspondence information.
We can rewrite L M using the joint adjacency matrix:
[Equation]
where Φ is the uniﬁed representation of the taxonomy entities and the knowledge graph entities, Φ =
[Equation]
J is the joint adjacency matrix, J
[Equation]
Eq.
(10) is the loss function for Laplacian eigenmaps [Belkin and Niyogi, 2003], which can be further derived into:
[Equation]
where the Laplacian L = D − J and D is a diagonal matrix with D ii = (cid:80)
j J ji .
It is easily seen that mapping all entities to zero (Φ = 0 ) minimizes the loss function L M , so an additional constraint Φ T D Φ = I needs to be added, where I is the identity matrix.
Theorem 1.
Minimizing L M with the constraint Φ T D Φ
= I to get the d -dimensional aligned embeddings (a ( | I T | + | I G | ) × d matrix Φ ) is equivalent to solving a generalized eigenvalue problem of the joint graph Laplacian: L v = λD v .
Proof.
When d = 1, Φ is a ( | I T | + | I G | )-vector, denoted as v 1 , then we have
[Equation]
Diﬀerentiating Eq.
(12) with respect to v 1 and λ , we have
[Equation]
So the optimal v 1 is a solution of the generalized eigenvalue problem: L v 1 = λD v 1 .
Multiplying both sides of Eq.
(13) by v T 1 and applying Eq.
(14), we have v T 1 L v 1 = λ .
To minimize L M = v T 1 L v 1 , we need to get the smallest nonzero eigenvalue λ 1 , then its corresponding eigenvector v 1 is the optimal solution.
When d > 1, Φ can be written as Φ =
[ v 1 , v 2 , . .
. , v d ].
The optimization problem is
[Equation]
Applying the same technique above, we have the minimal L M =
(cid:80) i λ i , where λ
i , for i = 1 , . . .
, d , are the d smallest nonzero eigenvalues sorted in ascending order: 0 < λ 1 ≤ · · · ≤ λ d , while their corresponding eigenvectors v 1 , . .
. , v d are the optimal solutions.
B.2 The Symmetry of J
Theorem 2.
The joint adjacency matrix J =
[Equation]
is symmetric.
Proof.
Recall the construction of the adjacency matrices A T and A G .
A T ij = 1 if and only if entity i is among the k nearest neighbors of entity j or j is among the k nearest neighbors of i according to I T , and the same with A G .
Thus A T and A G are symmetric.
Then the similarity matrices are constructed through Eq.
We have S T ij = exp(
− d h
( I T i , I T j ) t ) · A T ij = exp( − d h ( I T i , I T j ) t ) ·
A T ji = S T ji , similarly S G ij =
S G ji , so S T and S G are symmetric.
Then J is also symmetric.
B.3 The Symmetry and Positive Semi-Deﬁniteness of L
Theorem 3.
The Laplacian L = D − J is symmetric and positive semideﬁnite.
Proof.
First, L is symmetric because D and J are both symmetric ( D is a diagonal matrix and J is proved in Appendix B.2).
Next, from Eqs.
(10) and (11), we have (cid:80) i,j (cid:107) Φ i − Φ j (cid:107) 2 J i,j = tr(Φ T L Φ), which shows that L is positive semi-deﬁnite.
B.4 The Solutions of Manifold Alignment
Theorem 4.
The generalized eigenvalue problem L v = λD v has a zero eigenvalue λ 0 = 0 .
Proof.
Since L = D − J , D ii = (cid:80) j J ji
= (cid:80) j J ij , then ( L · [1 , 1 , . .
, 1] T ) i = (cid:80) j L ij =
D ii − (cid:80) j
( J ij )
= 0, that is, L · [1 , 1 , . . .
, 1] T = 0 , so λ
0 = 0.
Remark.
Combining Appendix B.1 and the fact that L is positive semi-deﬁnite (Appendix
B.3), the solutions of manifold alignment using Laplacian eigenmaps are v 1 , . .
. , v d , which are the corresponding eigenvectors of the d smallest nonzero eigenvalues: 0 = λ 0 < λ 1 ≤ . . .
λ
d .
Recall that Φ = (cid:20) φ I T
( I T )
φ I G ( I G ) (cid:21) =
[ v 1 , v 2 , . .
. , v d ] , so the aligned embedding vectors for I T and I G in the shared manifold are
[Equation]
C. δ -Hyperbolicity
The δ -hyperbolicity of a graph G [Gromov, 1987] is deﬁned as follows.
Deﬁnition 1.
Let a, b, c, d be vertices of the graph G .
Let S 1 , S 2 and S 3 be
[Equation]
Suppose M 1 and M 2 are the two largest values among S 1 , S 2 , S 3 and M 1 ≥ M 2 .
Deﬁne hyp ( a, b, c, d ) =
M 1 − M 2 .
Then the δ -hyperbolicity of G is deﬁned as
[Equation]
That is, δ ( G ) is the maximum of hyp over all possible 4 -tuples ( a, b, c, d ) divided by 2 .
D. More Experiments D.1 Hyperparameters
For pretraining of our approach, we learn the n -dimensional hyperbolic embeddings of the training taxonomy and the n -dimensional TransE embeddings of YAGOfacts, for n = { 5 , 10 , 20 , 50 , 100 } .
In manifold alignment, we project the embeddings of the two entity sets into 10-dimensional Euclidean space.
We use 5 nearest neighbors to construct A T and A G .
We assume the heat kernel parameter in Eq.
(4) to be + ∞ , i.e., S T =
A T , S G =
A G .
The coeﬃcient µ in L M Eq.
(5) is set as 0 .
The number of neighbors in k -nn when connecting the taxonomy entities I T and knowledge graph entities I n is 5.
The weights of edges in
E n for training the new taxonomy are 0 .
For pretraining and retraining, the epochs of hyperbolic embeddings and TransE are 500.
The hyperparameters of baselines are also tuned on the knowledge base combining
YAGOwordnet and YAGOfacts in 50-dimension by grid search, which are given in Table
[Table]
and YAGOfacts in 50-dimension.
We use the OpenKE repository 2 to train TransE, ComplEx, and RotatE. For TransC,HAKE, 4 JOIE, 5 MurP 6 , AttH 7 and HyperKA 8 , we use their public codes.
D.2 Evaluation
Our evaluation closely follows the setting of [Nickel and Kiela, 2017, 2018], which infers the hierarchies from distances in the embedding space.
Speciﬁcally, for each test entity u and the ground truth edge ( u, v ), we compute the distance between the embeddings d h ( u , v ) and rank it among the distances of all unobserved edges for u : { d h ( u , v (cid:48) ) :
( u, v (cid:48) ) / ∈ Training } .
We then report the following evaluation metrics of the rankings.
Denote I test as the test entity set.
Let NE u
= { v 1 , v 2 , . .
. , v | NE u | } be the set of the ground truth types of entity u .
https://github.com/thunlp/OpenKE 3. https://github.com/davidlvxin/TransC 4.
https://github.com/MIRALab-USTC/KGE-HAKE 5.
https://github.com/JunhengH/joie-kdd 6.
https://github.com/ibalazevic/multirelational-poincare 7.
https://github.com/HazyResearch/KGEmb 8.
https://github.com/nju-websoft/HyperKA
Test entity
Neighbors
Prediction
TransE
Eric Clapton
Brian Eno (artist) organism Smokey Robinson (artist) causal agent Chet Atkins (artist) living thing
GeoAlign
Eric Clapton
James Brown (artist) artist * Chet Atkins (artist) creator Willie Nelson (artist) person
TransE
Neil Young
Tony Banks (artist) organism Robbie Williams (artist) causal agent Glen Campbell (artist) property
GeoAlign
Neil Young
Moby (artist) creator Elton John (artist) artist
* Bj¨orn Ulvaeus (artist) person
TransE
Neil Gaiman
Stephen King (21st-century American novelists)
American male writers Gene Wolfe (fantasy writers)
American novelists Jonathan Lethem (American male essayists) 20th-century American novelists
GeoAlign
Neil Gaiman
Nikolai Gogoln (mythopoetic writers)
North American writers * Drago Janˇcar (Slovenian novelists) non-ﬁction writers * Charles Stross (Scottish science ﬁction writers)
English screenwriters *
TransE
Hannah Arendt
Judith Butler (20th-century women writers)
Translators Walter Benjamin (Franz Kafka scholars)
American novelists Karl Jaspers (Philosophers of technology)
American ﬁction writers
GeoAlign
Hannah Arendt
Milan Kundera (21st-century French novelists)
People in literature Horace (Golden Age Latin writers)
Writers Imre Lakatos (20th-century Hungarian writers)
European writers *
Mean average precision (MAP).
The average precision (AP) is a way to summarize the precision-recall curve into a single value representing the average of all precisions and the MAP score is calculated by taking the mean AP over all classes.
For an entity u , from the learned embeddings, we can obtain the types closest to its embedding u .
Let R u,v i be the smallest set of such types that contains v i (the i -th ground truth type of u ).
Then
the
MAP is deﬁned as:
[Equation]
P recision ( R u
[Equation]
v i ) .
Mean reciprocal rank (MRR).
The MRR is a statistic measure for evaluating a list of possible responses to a sample of queries, ordered by the probability of correctness.
For an entity u , from the learned embeddings, we can rank its distances with the types from the smallest to the largest.
Let rank v
i be the rank of v i (the i -th ground truth type of u ).
Then the MRR is deﬁned as:
[Equation]
The proportion of correct types that rank no larger than N (Hits@ N ).
Hits@
N measures whether the top N predictions contain the ground truth labels.
For an entity u , from the learned embeddings, we can obtain the set of N types closest to its embedding u , denoted as R Nu .
Then the Hits@ N is deﬁned as:
[Equation]
where I ( | R Nu ∩ NE u | ≥ 1) is the indicator function.
D.3 Case Study
Our manifold alignment is on top of the pretrained TransE model and the hyperboloid embeddings.
To have a more intuitive sense about the manifold alignment, in Table 9, we give some examples of the nearest neighbors of the test entity in the aligned manifold as well as the top 3 types predicted by GeoAlign.
We also present the corresponding items of
TransE as a comparison.
For TransE, the nearest neighbors are from its embedding space.
The examples Eric Clapton and Neil Young are from YAGOwordnet while Neil Gaiman and Hannah Arendt are from wikiObjects.
As we expect,
TransE and manifold alignment which leverages TransE both ﬁnd the reasonable nearest neighbors.
When looking into the top 3 predictions of Eric Clapton and Neil Young , we see that although TransE predicts correctly, it cannot get the ﬁne-grained types.
The most conﬁdent predictions of TransE are usually the general and rough types such as organism , living thing , while GeoAlign successfully predict them as artist .
For the test entities Neil Gaiman and Hannah Arendt , which come from the more ﬁne-grained and massive taxonomy wikiObjects, TransE cannot predict correctly at the top 3 predictions, but GeoAlign successfully obtains the correct ﬁne-grained types.
The results show that GeoAlign not only outputs accurate predictions but also captures the hierarchy structure to some extent.