SEEK : Segmented Embedding of Knowledge Graphs (2020)

Abstract: In recent years, knowledge graph embedding becomes a pretty hot research topic of artificial intelligence and plays increasingly vital roles in various downstream applications, such as recommendation and question answering. However, existing methods for knowledge graph embedding can not make a proper trade-off between the model complexity and the model expressiveness, which makes them still far from satisfactory. To mitigate this problem, we propose a lightweight modeling framework that can achieve highly competitive relational expressiveness without increasing the model complexity. Our framework focuses on the design of scoring functions and highlights two critical characteristics: 1) facilitating sufficient feature interactions; 2) preserving both symmetry and antisymmetry properties of relations. It is noteworthy that owing to the general and elegant design of scoring functions, our framework can incorporate many famous existing methods as special cases. Moreover, extensive experiments on public benchmarks demonstrate the efficiency and effectiveness of our framework. Source codes and data can be found at https://github.com/Wentao-Xu/SEEK .

## introduction learning embeddings for a knowledge graph (kg) is a vital task in artificial intelligence (ai) and can benefit many downstream applications, such as personalized recommendation @xcite and question answering @xcite . in general, a kg stores a large collection of entities and inter-entity relations in a triple format, (h, r, t), where h denotes the head entity, t represents the tail entity, and r corresponds to the relationship between h and t. the goal of knowledge graph embedding (kge) is to project massive * corresponding author. although recent years witnessed tremendous research efforts on the kge problem, existing research did not make a proper trade-off between the model complexity (the number of parameters) and the model expressiveness (the performance in capturing semantic information). to illustrate this issue, we categorize existing research into two categories. the first category of methods prefers the simple model but suffers from poor expressiveness. some early kge methods, such as transe @xcite and distmult @xcite , fell into this category. it is easy to apply these methods to large-scale real-world kgs, but their performance in capturing semantic information (such as link prediction) is far from satisfactory. in contrast, the second category pursues the excellent expressiveness but introduces much more model parameters and tensor computations. typical examples include transh @xcite , transr @xcite , transd @xcite , single distmult @xcite , conve @xcite and inter-acte @xcite . however, as pointed out by @xcite , the high model complexity often leads to poor scalability, which is prohibitive in practice because real-world kgs usually contain massive triples. to address these drawbacks of existing methods, in this paper, we propose a light-weight framework for kge that achieves highly competitive expressiveness without the sacrifice in the model complexity. next, we introduce our framework from three aspects: 1) facilitating sufficient feature interactions, 2) preserving various necessary relation properties, 3) designing both efficient and effective scoring functions. first, to pursue high expressiveness with the reasonable model complexity, we need to facilitate more sufficient feature interactions given the same number of parameters. specifically, we divide the embedding dimension into multiple segments and encourage the interactions among different segments. in this way, we can obtain highly expressive representations without increasing model parameters. accordingly, we name our framework as segmented embedding for kgs (seek). second, it is crucial to preserve different relation properties, especially the symmetry and the antisymmetry. we note that some previous research did not preserve the symmetry or the antisymmetry and thus obtained inferior performance @xcite @xcite . similar to the recent advanced models @xcite @xcite , we also pay close attention to the modeling support of both symmetric and antisymmetric relationships. third, after an exhaustive review of the literature, we find that one critical difference between various kge methods lies in the design of scoring functions. therefore, we dive deeply into designing powerful scoring functions for a triple (h, r, t). specifically, we combine the above two aspects (facilitating feature interactions and preserving various relation properties) and develop four kinds of scoring functions progressively. based on these scoring functions, we can specify many existing kge methods, including distmult @xcite , hoie @xcite , and com-plex @xcite , as special cases of seek. hence, as a general framework, seek can help readers to understand better the pros and cons of existing research as well as the relationship between them. moreover, extensive experiments demonstrate that seek can achieve either stateof-the-art or highly competitive performance on a variety of benchmarks for kge compared with existing methods. in summary, this paper makes the following contributions. ## related work we can categorize most of the existing work into two categories according to the model complexity and the model expressiveness. the first category of methods is the simple but lack of expressiveness, which can easily scale to large knowledge graphs. this kind of methods includes transe @xcite and dist-mult @xcite . transe uses relation r as a translation from a head entity h to a tail entity t for calculating their embedding vectors of (h, r, t); distmult utilizes the multi-linear dot product as the scoring function. the second kind of work introduces more parameters to improve the expressiveness of the simple methods. transh @xcite , transr @xcite , transd @xcite , and itransf @xcite are the extensions of transe, which introduce other parameters to map the entities and relations to different semantic spaces. the single distmult @xcite increases the embedding size of the dist-mult to obtain more expressive features. besides, proje @xcite , conve @xcite and interacte @xcite leverage neural networks to capture more feature interactions between embeddings and thus improves the expressiveness. however, these neural networkbased methods would also lead to more parameters since there are many parameters in the neural network. although the second kind of methods has a better performance compared with simple methods, they are difficult to apply to real-world kgs due to the high model complexity (a large number of parameters). compared with the two types of methods above, our seek can achieve high expressiveness without increasing the number of model parameters. ## seek briefly speaking, we build seek by designing scoring functions, which is one of the most critical components of various existing kge methods, as discussed in the related work. during the procedure of designing scoring functions, we progressively introduce two characteristics that hugely contribute to the model expressiveness: 1) facilitating sufficient feature interactions; 2) supporting both symmetric and antisymmetric relations. in this way, seek enables the excellent model expressiveness given a light-weight model with the same number of parameters as some simple kge counterparts, such as transe @xcite and distmult @xcite . ## experimental evaluation in this section, we present thorough empirical studies to evaluate and analyze our proposed seek framework. we first introduce the experimental setting. then we evaluate our seek framework on the task of link prediction. then, we study the influence of the number of segments k to the seek framework, and present the case studies to demonstrate why our seek framework has high effectiveness. ## conclusion and future work in this paper, we propose a lightweight kge framework (seek) that can improve the expressiveness of embeddings without increasing the model complexity. to this end, our framework focuses on designing scoring functions and highlights two critical characteristics: 1) facilitating sufficient feature interactions and 2) preserving various relation properties. besides, as a general framework, seek can incorporate many existing models, such as dist-mult, complex, and hole, as special cases. our extensive experiments on widely used public benchmarks demonstrate the efficiency, the effectiveness, and the robustness of seek. in the future, we plan to extend the key insights of segmenting features and facilitating interactions to other representation learning problems.