Complex Hyperbolic Knowledge Graph Embeddings with Fast F ourier Transform (2022)

Abstract: The choice of geometric space for knowledge graph (KG) embeddings can have significant effects on the performance of KG completion tasks. The hyperbolic geometry has been shown to capture the hierarchical patterns due to its tree-like metrics, which addressed the limitations of the Euclidean embedding models. Recent explorations of the complex hyperbolic geometry further improved the hyperbolic embeddings for capturing a variety of hierarchical structures. However, the performance of the hyperbolic KG embedding models for non-transitive relations is still unpromising, while the complex hyperbolic embeddings do not deal with multi-relations. This paper aims to utilize the representation capacity of the complex hyperbolic geometry in multi-relational KG embeddings. To apply the geometric transformations which account for different relations and the attention mechanism in the complex hyperbolic space, we propose to use the fast Fourier transform (FFT) as the conversion between the real and complex hyperbolic space. Constructing the attention-based transformations in the complex space is very challenging, while the proposed Fourier transform-based complex hyperbolic approaches provide a simple and effective solution. Experimental results show that our methods outperform the baselines, including the Euclidean and the real hyperbolic embedding models.

## introduction knowledge graph (kg) representation learning is important to the kg inference as well as the downstream tasks @xcite . it has been noticed that the embedding space has significant effects on the performance of kg completion tasks. previous works have proposed the kg embedding models in euclidean space @xcite @xcite , complex euclidean space @xcite , hyperbolic space @xcite ; embedding space hierarchical patterns multi-relation properties euclidean complex euclidean hyperbolic complex hyperbolic complex hyperbolic + fourier transform too flat and narrow to represent hierarchical patterns. too flat and narrow to represent hierarchical patterns. improve the euclidean embeddings a lot; limitations on hierarchies that deviate from tree metrics. further improve the hyperbolic embeddings to handle various and flexible hierarchical structures. maintain the representation capacity for various hierarchical structures. capture some relation properties. capture more relation properties. capture transitivity well; limitations on other relation properties. capture transitivity very well; not applicable to multi-relations. extend to multirelations and achieve good results. figure 1 : the summary of embedding spaces for hierarchical patterns and multi-relation properties. @xcite . 1 these models learn the embeddings of the kg entities in the selected geometric spaces and parameterize the relation representations as the geometric transformations, such as translation, rotation, matrix multiplication, etc. the euclidean and complex euclidean embedding models can capture relation properties including symmetry/anti-symmetry, inversion, and composition, but they cannot handle the transitive relations such as hypernymy. generally, the transitive relation forms a tree-like structure, for which hyperbolic geometry has a more powerful representation capacity than euclidean geometry because the hyperbolic space can be regarded as a continuous approximation to trees @xcite . however, most real-world graphs with transitivity do not necessarily form exact tree structures since the transitive relations can lead to a globally hierarchical structure with varying local structures, such as multitree structures @xcite and taxonomies @xcite . thus, the hyperbolic geometry which resembles tree metrics still has limitations on capturing various and flexible hierarchical structures. to tackle the limitation of hyperbolic embeddings, a recent work @xcite proposed to explore the complex hyperbolic geometry to learn the embeddings of hierarchical graphs. due to the variable negative curvature (goldman, 1999), the complex hyperbolic space is more flexible in handling varying structures while the tree-like properties are still retained. despite the remarkable improvements in single transitive relation inference, the complex hyperbolic geometry has not been utilized for multirelational embeddings. in this paper, we are motivated to make use of the complex hyperbolic geometry's representation superiority in kgs. there are two main challenges in extending the complex hyperbolic embeddings to multiple relations. first, the geometric transformations in complex hyperbolic geometry are complicated and challenging to optimize due to the numerical instabilities, making it difficult to apply the complex geometric transformations for different relations. second, it is hard to build the neural network unit or layer in the complex domain. missing the complex attention mechanism would restrict the parameterization capability and make the complex domain-based model difficult to generalize to further downstream tasks. to address the above problems, we propose a complex hyperbolic kg embedding approach with the fast fourier transform. our approach can utilize the representation capacity of the complex hyperbolic geometry as well as the well-developed attention-based geometric transformations as relation parameterization, while we borrow the fast fourier transform (fft) and inverse fast fourier transform (ifft) to provide the conversion between the real and complex hyperbolic space. we regard the complex hyperbolic embeddings in the unit ball model (a projective geometry-based model to identify the complex hyperbolic space) (goldman, 1999) and the hyperbolic embeddings in the poincaré ball model (a model of the real hyperbolic space) @xcite as frequency domain and spatial domain respectively. then fft and ifft enable us to convert the embeddings between the two geometric spaces, accomplishing the leverage of real hyperbolic transformations to the complex hyperbolic model. the framework is simple and effective in learning the complex hyperbolic kg representations. figure 1 summarizes the comparison among embedding spaces for hierarchical patterns and multi-relation properties. in experiments, we evaluate our approach on the kg link prediction task with two popular benchmarks-wn18rr @xcite and fb15k-237 @xcite . empirical results show that our fourier transform-based complex hyperbolic kg embedding approach outperforms the baseline models in other geometric spaces. the code and data of our work are available at https://github.com/hkust-knowcomp/ complexhyperbolickge . complex euclidean kg embeddings. the follow-up works @xcite @xcite extended the traditional euclidean models to complex hyperbolic geometry. specifically, complex @xcite found that the hermitian dot product can effectively capture anti-symmetric relations while retaining the efficiency benefits of the dot product. rotate @xcite defined each relation as a rotation in the complex vector space to infer various relation patterns (symmetry/antisymmetry, inversion, composition). the effectiveness of these models revealed the potential of the complex geometry. hyperbolic embeddings. in recent years, the hyperbolic space attracted much attention for representation learning since it can naturally characterize tree structures. the hyperbolic embedding methods have developed from the single transitive relation graphs @xcite @xcite to multi-relational kgs @xcite . murp @xcite embedded the hierarchical multi-relational data in the poincaré ball model and learned relation-specific parameters by möbius operations. the state-of-the-art hyperbolic kg embedding models are a series of hyperbolic transformation-based models refh, roth, and atth @xcite , which utilize the geometric tree-like property to capture the hierarchical structure naturally while using different geometric transformations as well as attention mechanism to parameterize other relation properties. ## approach given the kg with entity set @xmath0, relation set @xmath1, and triplet set @xmath2, r, t)|h, @xmath3, @xmath4, the link prediction task aims to predict the tail entity t for each test query (h, r). to train the models for a higher-quality inference, we learn the entity embeddings {ej}m @xmath5, while parameterizing the relations by the hyperbolic transformations roth, refh, and the hyperbolic attention-based model atth @xcite . we construct the conversion between the two geometries through fourier transform. figure 2 presents the overview of our framework. in this section, we first present the attention-based hyperbolic transformations as relation parameterization, then introduce the fourier transform as the conversion between complex hyperbolic domain and hyperbolic domain, followed by the details of our framework. ## experiments in this section, we evaluate our approaches on the kg link prediction task. we show that our complex hyperbolic embedding models outperform the baseline methods based on other geometric spaces. ## conclusion and future work in this work, we explore the complex hyperbolic geometry for multi-relational kg embeddings. the whole framework utilizes the fourier transform as the efficient conversion between geometric spaces. with the aid of the fourier transform, the complex hyperbolic embeddings can be transformed into the real domain and be capable of applying real hyperbolic transformations, which enables our approach to take the advantages of both the powerful complex hyperbolic geometry and the attention-based real hyperbolic transformations. experiments show that the fourier transform-based complex hyperbolic embedding models can effectively learn the kg embeddings and outperform the baseline models of other spaces in the link prediction task. we believe our proposed approach not only provides a novel and interesting representation learning framework for kgs but also potentially inspires the learning algorithms for more general multi-relational data and contributes to improvements on more downstream tasks.