Low-Dimensional Hyperbolic Knowledge Graph Embeddings (2020)

Abstract: Knowledge graph (KG) embeddings learn low- dimensional representations of entities and relations to predict missing facts. KGs often exhibit hierarchical and logical patterns which must be preserved in the embedding space. For hierarchical data, hyperbolic embedding methods have shown promise for high-fidelity and parsimonious representations. However, existing hyperbolic embedding methods do not account for the rich logical patterns in KGs. In this work, we introduce a class of hyperbolic KG embedding models that simultaneously capture hierarchical and logical patterns. Our approach combines hyperbolic reflections and rotations with attention to model complex relational patterns. Experimental results on standard KG benchmarks show that our method improves over previous Euclidean- and hyperbolic-based efforts by up to 6.1% in mean reciprocal rank (MRR) in low dimensions. Furthermore, we observe that different geometric transformations capture different types of relations while attention- based transformations generalize to multiple relations. In high dimensions, our approach yields new state-of-the-art MRRs of 49.6% on WN18RR and 57.7% on YAGO3-10.

## introduction knowledge graphs (kgs), consisting of (head entity, relationship, tail entity) triples, are popular data structures for representing factual knowledge to be queried and used in downstream applications such as word sense disambiguation, question answering, and information extraction. real-world kgs such as yago @xcite or wordnet @xcite are usually incomplete, so a common approach to predicting missing links in kgs is via embedding into vector spaces. embedding methods learn representations of entities and relationships that preserve the information found in the graph, and have achieved promising results for many tasks. relations found in kgs have differing properties: for example, (michelle obama, married to, barack obama) is symmetric, whereas hypernym relations like (cat, specific type of, feline), are not (figure 1 ). these distinctions present a challenge to embedding methods: preserving each type of behavior requires producing a different geometric pattern in the embedding space. one popular approach is to use extremely high-dimensional embeddings, which offer more flexibility for such patterns. however, given the large number of entities found in kgs, doing so yields very high memory costs. for hierarchical data, hyperbolic geometry offers an exciting approach to learn low-dimensional embeddings while preserving latent hierarchies. hyperbolic space can embed trees with arbitrarily low distortion in just two dimensions. recent research has proposed embedding hierarchical graphs into these spaces instead of conventional euclidean space @xcite . however, these works focus on embedding simpler graphs (e.g., weighted trees) and cannot express the diverse and complex relationships in kgs. we propose a new hyperbolic embedding ap-proach that captures such patterns to achieve the best of both worlds. our proposed approach produces the parsimonious representations offered by hyperbolic space, especially suitable for hierarchical relations, and is effective even with lowdimensional embeddings. it also uses rich transformations to encode logical patterns in kgs, previously only defined in euclidean space. to accomplish this, we (1) train hyperbolic embeddings with relation-specific curvatures to preserve multiple hierarchies in kgs; we evaluate the performance of our approach, atth, on the kg link prediction task using the standard wn18rr @xcite , fb15k-237 @xcite and @xcite benchmarks. (1) in low (32) dimensions, we improve over euclidean-based models by up to 6.1% in the mean reciprocical rank (mrr) metric. in particular, we find that hierarchical relationships, such as wordnet's hypernym and member meronym, significantly benefit from hyperbolic space; we observe a 16% to 24% relative improvement versus euclidean baselines. (2) we find that geometric properties of hyperbolic isometries directly map to logical properties of relationships. we study symmetric and anti-symmetric patterns and find that reflections capture symmetric relations while rotations capture anti-symmetry. (3) we show that attention based-transformations have the ability to generalize to multiple logical patterns. for instance, we observe that atth recovers reflections for symmetric relations and rotations for the antisymmetric ones. in high (500) dimensions, we find that both hyperbolic and euclidean embeddings achieve similar performance, and our approach achieves new stateof-the-art results (sota), obtaining 49.6% mrr on wn18rr and 57.7% yago3-10. our experiments show that trainable curvature is critical to generalize hyperbolic embedding methods to highdimensions. finally, we visualize embeddings learned in hyperbolic spaces and show that hyperbolic geometry effectively preserves hierarchies in kgs. ## related work previous methods for kg embeddings also rely on geometric properties. improvements have been obtained by exploiting either more sophisticated spaces (e.g., going from euclidean to complex or hyperbolic space) or more sophisticated operations (e.g., from translations to isometries, or to learning graph neural networks). in contrast, our approach takes a step forward in both directions. ## problem formulation and background we describe the kg embedding problem setting and give some necessary background on hyperbolic geometry. ## methodology the goal of this work is to learn parsimonious hyperbolic embeddings that can encode complex logical patterns such as symmetry, anti-symmetry, or inversion while preserving latent hierarchies. our model, atth, (1) learns kg embeddings in hyperbolic space in order to preserve hierarchies (section 4.1), (2) uses a class of hyperbolic isometries parameterized by compositions of givens transformations to encode logical patterns (section 4.2), (3) combines these isometries with hyperbolic attention (section 4.3). we describe the full model in section 4.4. ## experiments in low dimensions, we hypothesize (1) that hyperbolic embedding methods obtain better representations and allow for improved downstream performance for hierarchical data (section 5.2). ( 2) we expect the performance of relation-specific geometric operations to vary based on the relation's logical patterns (section 5.3). (3) in cases where the relations are neither purely symmetric nor antisymmetric, we anticipate that hyperbolic attention outperforms the models which are based on solely reflections or rotations (section 5.4). finally, in high dimensions, we expect hyperbolic models with trainable curvature to learn the best geometry, and perform similarly to their euclidean analogues (section 5.5). ## conclusion we introduce atth, a hyperbolic kg embedding model that leverages the expressiveness of hyperbolic space and attention-based geometric transformations to learn improved kg representations in low-dimensions. atth learns embeddings with trainable hyperbolic curvatures, allowing it to learn the right geometry for each relationship and generalize across multiple embedding dimensions. atth achieves new sota on wn18rr and yago3-10, real-world kgs which exhibit hierar-chical structures. future directions for this work include exploring other tasks that might benefit from hyperbolic geometry, such as hypernym detection. the proposed attention-based transformations can also be extended to other geometric operations.