T ran SHER : Translating Knowledge Graph Embedding with Hyper-Ellipsoidal Restriction (2022)

Abstract: Knowledge graph embedding methods are important for the knowledge graph completion (or link prediction) task.One state-of-the-art method, PairRE, leverages two separate vectors to model complex relations (i.e., 1-to-N, N-to-1, and N-to-N) in knowledge graphs. However, such a method strictly restricts entities on the hyper-ellipsoid surfaces which limits the optimization of entity distribution, leading to suboptimal performance of knowledge graph completion. To address this issue, we propose a novel score function TranSHER, which leverages relation-specific translations between head and tail entities to relax the constraint of hyper-ellipsoid restrictions. By introducing an intuitive and simple relation-specific translation, TranSHER can provide more direct guidance on optimization and capture more semantic characteristics of entities with complex relations. Experimental results show that TranSHER achieves state-of-the-art performance on link prediction and generalizes well to datasets in different domains and scales. Our codes are public available athttps://github.com/yizhilll/TranSHER.

## introduction knowledge graph is proposed to structurally store human knowledge when the development of computer science has brought exponentially growing digitized information. knowledge graphs have been widely adopted in important applications, such as semantic parsing @xcite , question generation and answering @xcite @xcite , and information retrieval @xcite . however, due to the expanding nature and difficulties of construction, knowledge graphs often suffer from incompleteness. thus, knowledge graph completion (or link prediction) becomes a task of great importance from both a research and application perspective. to construct a knowledge graph, entities and relations extracted from facts are treated as nodes and edges. a single fact in this graph is represented as a directed relation-specific link between two entities, which is denoted as a triplet like @xcite . since knowledge graphs usually contain a large number of entities and relations, knowledge graph embeddings are proposed to efficiently learn the representation of graphs and accomplish the link prediction task with score functions. intuitions behind the design of score functions can be summarized as two important principles: 1) logical reasoning relation patterns such as symmetry (antisymmetry), inversion, and composition; 2) statistically categorized relation types like 1-to-1, 1-to-n, nto-1, and n-to-n, where the latter three are called complex relations. inspired by word2vec @xcite , transe @xcite represents knowledge graphs with relation-specific translations between head and tail entities to model the connections, where the score function works upon the translational distance ∥e h +r -e t ∥. following this design, certain works @xcite @xcite improve transe by conducting relation-specific transformations on the entities be-fore calculating distance. certain works @xcite @xcite further apply the principles of modeling relation patterns and complex relations to refine the design of score functions. among these methods, the recently proposed pairre @xcite uses separate relation vectors for head and tail entities to better model the complex relations and outperforms preceding methods. however, after our preliminary exploration, we notice that pairre imposes a strong restriction that fixes the l 2 norm of the entity vector ⃗ e, and scales it with coefficients along all dimensions. under such a restriction, the entities are essentially distributed on the surface of hyper-ellipsoids, where the foci are exactly laying on the axes. for the same reason, entity embeddings are strictly limited and can only be optimized around the hyper-ellipsoidal surface. in other words, entity distribution can only move along an arc path to match the true connections, as fig. 1a shows, which may impose the close entity embeddings entangled and bring difficulties to the modeling. to tackle the aforementioned challenges, we propose transher which has a novel score function that leverages relation-specific translations between head and tail entities to relax the constraint of hyper-ellipsoid restrictions. we hypothesize that introducing translations for entities of pairre can simplify the multi-relation modeling of entities on hyper-ellipsoids by providing an extra degree of freedom and thus improve the distribution learning of entities. in order to model complex relations, transher first follows pairre to conduct mappings on head and tail entities separately. then, transher additionally performs relation-specific translations on entities while holding the hyperellipsoidal restriction. take fig. 1b as an example: the mapped head entities (green) are moved closer to the tail entities (orange) without requiring obvious changes of the entities within the cluster and form a better distribution for relational modeling because the translational distance provides more direct guidance in the score function. the relation-specific translation in our score function allows more flexible optimization by relaxing the arc path constraint in pairre brought by the hyperellipsoidal mapping, leading to a better distribution of entities and knowledge graph completion. moreover, transher can better model the semantic characteristics of entities and utilize them to improve entity retrieval for complex relations. the semantic characteristics usually decide the categories of entities (e.g., people or companies); transher can model the connected structure between entities of different categories more accurately in complex relations. for those complex relation triplets, transher further enhances link prediction by retrieving multiple candidate entities in the correct category. for example, for a tail prediction query (film, produced_by, ?), transher ranks the names of film producers at the front while other models may be confused by the related entertainment company or studio (cf. §5.6). in addition, while achieving qualitative embeddings, transher can further maintain the ability to model important relation patterns, namely, symmetry (antisymmetry), inversion, and composition under certain constraints (cf. §4). with such ability, transher has the potential to be generalized well on datasets from different knowledge domains in different real-world settings. we conduct comprehensive experiments across five datasets from different domains, which demonstrate the effectiveness of transher. impressively, transher has achieved substantial improvement compared with the best baseline: the mrr increase has reached up to 4.6% on yago37 and 3.2% on ogbl-wikikg2. we also conduct many analytical experiments to study how translations of transher behave and enhance knowledge graph completion. some case studies further demonstrate the superiority of our approach. ## related work knowledge graph embedding methods are proposed to model the intrinsic properties of facts and to conduct knowledge graph completion. to further complete knowledge graphs, these methods use designed score functions to model complex relations and various patterns. by measuring the relation-related distance between entities, kge models predict the probabilities of given triplet queries (e h , r, ?) or (?, r, e t ) to complete the graph. such models are characterized by their corresponding score functions and the embeddings are usually optimized with gradient descent algorithms. distance-based score functions model the triplet facts by calculating distances between entity embeddings in the euclidean space. proposed by transe @xcite , one popular practice is to conduct a relation-specific translation on the given entity before distance calculation, i.e., let e h + r ≈ e t in the case where the fact holds. such a translational principle empowers the models to conduct knowledge graph completion on largescale datasets while maintaining their performance. many score functions, such as transh, transr, and manifolde @xcite @xcite , followed this principle of translation and achieved fair performances. however, ro-tate @xcite claims these extended works of transe are weak in modeling certain relation patterns and thus proposes a solution of modeling in the complex space. moreover, pairre @xcite argues that complex relations can be better modeled by separating relation vectors for heads and tails. our transher aims to provide a more effective model for complex relations by bridging the gap between translational distance-based models and the latest model pairre. semantic matching score functions aim to predict the existence of facts by measuring the semantic similarity among entities and the given relation in the same representation space. rescal @xcite introduces a bilinear function h ⊤ m r t to represent the similarity score but suffers from modeling complexity. some following work such as distmult, hole, and complex @xcite @xcite intend to simplify the model while preserving critical features. a more recent work seek @xcite generalizes the existing semantic matching score functions by segmenting the embedding to facilitate feature interactions, while keeping the same model size. in general, the semantic matching models struggled to distinguish similar entities and lack the ability to simultaneously model multiple relation patterns. our transher model tackles these challenges by proposing a novel score function which leverages relation-specific translations, yielding an effective improvement in modeling complex relations. ## problem formulation in this section, we describe the task definition and notations for better illustration. the set of known facts in a knowledge graph is represented by t , which includes triplets (e h , r, e t ). the notation (e h , r, e t ) ∈ t will be used when the fact holds. the set of entity e and the set of relation r are denoted as e and r. knowledge graph completion (also regarded as link prediction) aims to predict the missing links of knowledge graphs. specifically, given a triplet (e h , r, e t ′ ) or (e h ′ , r, e t ), a score function f r is required to output an existence probability of the triplet. since all the entities in e are provided in the training set, knowledge graph completion can also be regarded as a ranking task. for the true candidate entities, models are expected to assign higher rankings than false ones. for a given entity-relation query (e h , r, ?) or (?, r, e t ), there may exist multiple correct answers to complete the triplet, i.e., the quantities of those entities satisfy ∥{e according to the average heads per tail and tails per head counted through the dataset, relations in r can be categorized into 4 types: 1-to-1, 1-to-n, n-to-1, and n-to-n @xcite . relations can also be summarized by several significant patterns: symmetry/antisymmetry, inversion, and composition. the definitions are given as follows: ## conclusion we propose a novel knowledge graph embedding model transher for the link prediction task. tran-sher leverages relation-specific translation on entities with hyper-ellipsoidal restriction, which is explicitly encoded into the score function. by introducing the translation, transher can improve the optimization of entities distributed on hyperellipsoids and shows ingenuity in understanding semantic characteristics. moreover, we prove that transher preserves the ability to represent logical reasoning relation patterns. comprehensive experiments on different datasets show that transher has robust performance and improves complex relation modeling.