Hyperbolic Geometry is Not Necessary: Lightweight E uclidean-Based Models for Low-Dimensional Knowledge Graph Embeddings (2021)

Abstract: Recent knowledge graph embedding (KGE) models based on hyperbolic geometry have shown great potential in a low-dimensional embedding space. However, the necessity of hyperbolic space in KGE is still questionable, because the calculation based on hyperbolic geometry is much more complicated than Euclidean operations. In this paper, based on the state-of-the-art hyperbolic-based model RotH, we develop two lightweight Euclidean-based models, called RotL and Rot2L. The RotL model simplifies the hyperbolic operations while keeping the flexible normalization effect. Utilizing a novel two-layer stacked transformation and based on RotL, the Rot2L model obtains an improved representation capability, yet costs fewer parameters and calculations than RotH. The experiments on link prediction show that Rot2L achieves the state-of-the-art performance on two widely-used datasets in low-dimensional knowledge graph embeddings. Furthermore, RotL achieves similar performance as RotH but only requires half of the training time.

## introduction to represent entities and relations of knowledge graphs (kgs) in the semantic vector space, researchers have proposed various knowledge graph embedding (kge) models, which have shown great potential in knowledge graph completion and knowledge-driven applications @xcite . to achieve higher prediction accuracy, recent kge models usually use highdimensional embedding vectors up to 200 or even 500 dimensions @xcite . however, when facing large-scale kgs with millions of entities, high embedding dimensions would require prohibitive training costs and storage space @xcite . it hinders the practical application of kge models, especially in mobile smart devices. recently, low-dimensional kge models based on hyperbolic vector space have drawn some attention @xcite . the work of the first such model, murp, indicates that hyperbolic embeddings can capture hierarchical patterns in kgs and generate high-fidelity and parsimonious representations @xcite . to capture logical patterns in kgs, @xcite propose a series of hyperbolic kge models, including roth, refh, and atth @xcite . similar to the typical transe model @xcite ) that treats the relation as a translation operation between the head and tail entity vectors, the state-of-the-art roth model adjusts the head vector by the rotation and translation transformations to approach the tail vector in the hyperbolic space. although the above hyperbolic-based models outperform previous euclidean-based models in low-dimensional condition, the necessity of hyperbolic space in this task is still questionable. comparing a hyperbolic model with its euclidean-based variant, it is uncertain which parts of the modifi-cation will be vital. besides, despite theoretical support, the möbius matrix-vector multiplication and möbius addition operations in hyperbolic embeddings are far more complicated than euclidean multiplication and addition. as shown in fig. 1 , roth requires threefold more training time than its euclidean-based variant rote on two datasets. especially on large-scale knowledge graphs, the additional calculating cost caused by the complicated hyperbolic operations would make the problem much severer. facing these problems, we analyze the effective components in the hyperbolic kge models, and propose two lightweight "roth-like" models, rotl and rot2l, for low-dimensional knowledge graph embeddings. without the hyperbolic geometry, rotl eliminates the möbius matrix-vector multiplication and designs a new flexible addition operation to replace the möbius addition. to further improve the rotl's representation capability, the rot2l model utilizes two stacked rotationtranslation transformations in the euclidean space. benefiting from a specific parameterization strategy, rot2l requires fewer parameters and calculations than roth. we conduct extensive experiments on two widely-used datasets. the results show that rotl outperforms existing euclidean-based models in the 32-dimensional condition and only requires half of the training time of roth. rot2l obtains the state-of-the-art performance on the two datasets and outperforms roth in both prediction accuracy and training speed. according to ablation experiments, we prove the effectiveness of the flexible addition and the other significant modules in rot2l. we also verify our models in different embedding dimensions and analyze the performance difference between roth and our models in a relation-specific experiment. the rest of the paper is organized as follows. we discuss the background and definitions in sec. 2. sec. 3 introduces the technical details of rotl and rot2l models. sec. 4 reports the experimental studies and sec. 5 further discusses several experimental investigations. the related work is reviewed in sec. 6. finally, we offer some concluding remarks in sec. 7. ## background in this section, we briefly describe the preliminaries related to this work. ## the methodology the goal of this work is to design a high-efficiency low-dimensional kge model by extracting the effective components in the roth model and eliminating the redundancy. we find that roth performs noticeably well because of two reasons. the first reason is rotationtranslation transformation. as proved in previous research @xcite , this specific transforming operation can infer different relation patterns in the kg. the second reason is flexible normalization. all entity vectors in the hyperbolic space satisfy e 2 < 1 c before and after transformation, while the curvature c is relation-specific and self-adaptive. as the representation capability of a low-dimensional vector space is limited, the effect of flexible normalization would be more obvious. it explains why roth can outperform its euclidean-based variant rote in low-dimensional kge tasks. in this section, we first propose a lightweight model, called rotl, which remains the flexible normalization of roth and simplifies the complex hyperbolic operations. the details of rotl will be described in sec. 3.1. we further design the rot2l model using two stacked rotation-translation transformations. rot2l employs a novel parameterization strategy that can save half of parameters in the two-layer architecture, which is detailed in sec. 3.2. the architectures of the four models mentioned above are illustrated in fig. 2 . ## discussion in this section, we further discuss several important questions on the rotl and rot2l models. q1: which parts of predictions are improved in our models comparing with roth? we measure the link prediction performance of relation-specific triples on wn18rr, shown in ## conclusion the recently proposed hyperbolic-based models achieve great prediction accuracy in lowdimensional knowledge graph embeddings, but require complicated calculations for hyperbolic embeddings. in this paper, we analyze the effective components in those models and propose a lightweight variant based on euclidean calculations. after simplifying the möbius operations in roth, our proposed rotl model achieves a competitive performance, which saves half of the training time. using a two-layer stacked transformation, we further propose rot2l that outperforms the state-ofthe-art roth model in both prediction accuracy and training speed. these positive results encourage us to explore further research activities in the future. we will theoretically analyze the effectiveness of flexible normalization in the low-dimensional kge tasks. for the stacked transformations in rot2l, we will explore multiple-layer architectures and evaluate more different transformation forms. finally, we plan to apply our models on real-world knowledge graphs in different domains such as mobile healthcare, smart cities, and mobile e-commerce.