Block-Diagonal Orthogonal Relation and Matrix Entity for Knowledge Graph Embedding (2024)

Abstract: The primary aim of Knowledge Graph Embeddings (KGE) is to learn low-dimensional representations of entities and relations for predicting missing facts. While rotation-based methods like RotatE and QuatE perform well in KGE, they face two challenges: limited model flexibility requiring proportional increases in relation size with entity dimension, and difficulties in generalizing the model for higher-dimensional rotations. To address these issues, we introduce OrthogonalE, a novel KGE model employing matrices for entities and block-diagonal orthogonal matrices with Riemannian optimization for relations. This approach not only enhances the generality and flexibility of KGE models but also captures several relation patterns that rotation-based methods can identify. Experimental results indicate that our new KGE model, OrthogonalE, offers generality and flexibility, captures several relation patterns, and significantly outperforms state-of-the-art KGE models while substantially reducing the number of relation parameters.

## introduction the fundamental elements of knowledge graphs (kgs) are factual triples, each represented as (h, r, t), indicating a relationship r between head entity h and tail entity t. notable examples include freebase @xcite , yago @xcite , and wordnet @xcite . kgs have practical applications in various fields such as question-answering @xcite , information retrieval @xcite , recommender systems @xcite , and natural language processing @xcite , garnering our code is available at https://github.com/ yihuazhu111/orthogonale . addressing the inherent incompleteness of kgs, link prediction has become a pivotal area of focus. recent research @xcite has extensively leveraged knowledge graph embedding (kge) techniques, aiming to learn compact, low-dimensional representations of entities and relations. these approaches, marked by scalability and efficiency, have shown proficiency in modeling and deducing kg entities and relations from existing facts. recently, rotation-based kge like rotate @xcite and quate @xcite methods have achieved notable success in the field. ro-tate uses the hadamard product to multiply the real and imaginary components of the head entity embedding by the angle-based relation embedding, resulting in a 2d rotation effect within each unit. each unit consists of two elements representing the real and imaginary components. for example, an entity embedding with 500 dimensions consists of 250 units. the resulting vectors in each unit are then concatenated to form the tail entity embedding. the essence of this operation is the matrix multiplication between a 2d rotation matrix and the entity vector, followed by concatenation as well, as illustrated in fig. 1 , multiplying the relation matrix e 1 @xmath0: r 2×2 ) with the head entity vector e 1 @xmath1. here, we treat the concatenation operation as a block-diagonal arrangement of each 2d rotation matrix. similarly, quate extends rotate by using quaternions, which consist of one real and three imaginary components, achieving a 3d rotation effect through the multiplication of a 3d rotation matrix with the entity vector. for quate, we replace the 2d rotation matrix r 2×2 with a 3d rotation matrix r 3×3 as shown in fig. 1 . however, these approaches face two primary issues: lack of flexibility and generality, as illustrated in fig. 1 . first, the model's lack of flexibility necessitates increasing the size of the overall relation matrix ( n+l) ) to meet entity dimension requirements (e 1 @xmath2. for example, when the entity vector changes (e 1 @xmath3, the parameter increase is 900. however, the corresponding change in the relation ma- second, the model's lack of generality makes it challenging to explore high-dimensional rotational kge models due to the significant computational demands and the complexity of rotations in higher dimensions (b @xmath4, r 3×3 → r d×d , @xmath5, such as so(4), so(5), and so(10) 1 . this restricts the development of more generalized and higherdimensional rotation kge approaches. to overcome these two issues, we propose a highly general and flexible kge model named or-thogonale as shown in fig. 2 , and detailed notation is shown in we evaluate the new model on two kge datasets including wn18rr @xcite , fb15k-237 @xcite transe @xcite , which interprets relations as vector translations. transh @xcite , transr @xcite , and transd @xcite represent extensions of the translationbased method, building upon the foundational approach of transe. complex @xcite advances this by embedding kgs in a complex space and using the hermitian product for modeling antisymmetric patterns. inspired by complex, rotate @xcite then innovated by treating relations as rotations in a complex vector space, capable of capturing varied relation patterns like symmetry, antisymmetry, inversion, and commutative composition. following this, quate (zhang et al., 2019) employed quaternion operations (3d rotations) for even better expressiveness than ro-tate. dense (lu et al., 2022) employed various techniques for 3d rotation implementation and proposed that 3d rotation could handle the relation pattern of non-commutative composition. hopfe (bastos et al., 2021) seeks to employ so(4) rather than so(3) for kg representation, which is directly connected to the generality issue discussed in our research. we are also keen on investigating rotations in higher dimensions. nonetheless, progressing to so(5) or even so(10) poses substantial difficulties. in addition, our research builds upon existing work in the field. for instance, ote @xcite uses a compact block-diagonal orthogonal matrix, similar to our approach, to maintain ro-tate's relation patterns and complex relations. yet, our orthogonale model improves stability and performance with riemannian optimization, outperforming ote's gram-schmidt process. golde's @xcite universal orthogonal parameterization, derived from householder reflections, offers theoretical guarantees for dimensionality and geometry. however, our focus is on enhancing the generality and flexibility of the kge models, not just on capturing knowledge graph patterns and heterogeneity. in conclusion, considering the two major disadvantages of rotation-based methods mentioned in section 1, we need to refine our model to make it more general and flexible. optimization on the orthogonal manifold in optimization on the orthogonal manifold, transitioning from x t to x t+1 while remaining on the manifold necessitates a method known as retraction @xcite . prior research has effectively adapted several standard euclidean function minimization algorithms to riemannian manifolds. notable examples include gradient descent @xcite , second-order quasi-newton methods @xcite , and stochastic approaches @xcite , crucial in deep neural network training. meanwhile, riemannian optimization is often used for the orthogonal manifolds, and has made significant progress in deep learning, especially in cnns and rnns. @xcite innovatively substituted cnn's batch normalization layers with riemannian optimization on the grassmann manifold for parameter normalization. additionally, significant strides in stabilizing rnn training have been made by @xcite @xcite , through the application of riemannian optimization to unitary matrices. as this paper primarily focuses on kge, we do not delve deeply into riemannian optimization. instead, we utilize retraction with the exponential map for iterative optimization, as implemented in the geoopt library @xcite . ## problem formulation and background before presenting our approach, we introduce the kge problem and provide an overview of optimization on orthogonal manifolds. ## approach our approach is developed to acquire both a flexible and general kge model and ensure that this model can concurrently represent several relation patterns. this is achieved by employing matrices for entities and block-diagonal orthogonal matrices with riemannian optimization for relations. figure 2 illustrates the orthogonale approach, and ## experiment we expect that our proposed orthogonale model, employing matrices for entities and block-diagonal orthogonal matrices with riemannian optimization for relations, will outperform baseline models. also, we anticipate that orthogonale is a general and flexible kge model and can represent several relation patterns simultaneously. our goal is to validate these through empirical testing.