G reen KGC : A Lightweight Knowledge Graph Completion Method (2023)

Abstract: Knowledge graph completion (KGC) aims to discover missing relationships between entities in knowledge graphs (KGs). Most prior KGC work focuses on learning embeddings for entities and relations through a simple score function. Yet, a higher-dimensional embedding space is usually required for a better reasoning capability, which leads to larger model size and hinders applicability to real-world problems (e.g., large-scale KGs or mobile/edge computing). A lightweight modularized KGC solution, called GreenKGC, is proposed in this work to address this issue. GreenKGC consists of three modules: representation learning, feature pruning, and decision learning, to extract discriminant KG features and make accurate predictions on missing relationships using classifiers and negative sampling. Experimental results demonstrate that, in low dimensions, GreenKGC can outperform SOTA methods in most datasets. In addition, low-dimensional GreenKGC can achieve competitive or even better performance against high-dimensional models with a much smaller model size.

## introduction knowledge graphs (kgs) store human knowledge in a graph-structured format, where nodes and edges denote entities and relations, respectively. a (head entity, relation, tail entity) factual triple, denoted by (h, r, t), is a basic component in kgs. in many knowledge-centric artificial intelligence (ai) applications, such as question answering @xcite , information extraction @xcite , and recommendation @xcite , kg plays an important role as it provides explainable reasoning paths to predictions. however, most kgs suffer from the incompleteness when a model has fewer parameters, its performance is poorer. also, the larger dataset, yago3-10, demands more parameters than the smaller dataset, fb15k-237, to achieve satisfactory results. the requirement of high-dimensional embeddings for popular kge methods comes from the over-simplified scoring functions @xcite . thus, classification-based kgc methods, such as conve @xcite , aim to increase the reasoning capabilities in low dimensions by adopting neural networks (nns) as powerful decoders. as a result, they are more efficient in parameter scaling than kge models @xcite . however, nns demand longer inference time and more computation power due to their deep architectures. the long inference time of the classification-based methods also limits their applicability to some tasks that require real-time inference. recently, dualde @xcite applied knowledge distillation (kd) @xcite to train powerful low-dimensional embeddings. yet, it demands three stages of embedding training: 1) training high-dimensional kge, 2) training low-dimensional kge with the guidance of high-dimensional kge, and 3) multiple rounds of student-teacher interactions. its training process is time-consuming and may fail to converge when the embeddings are not well-initialized. here, we propose a new kgc method that works well under low dimensions and name it greenkgc. greenkgc consists of three modules: 1) representation learning, 2) feature pruning, and 3) decision learning. each of them is trained independently. in module 1, we leverage a kge method, called the baseline method, to learn high-dimensional entity and relation representations. in module 2, a feature pruning process is applied to the high-dimensional entity and relation representations to yield discriminant low-dimensional features for triples. in addition, we observe that some feature dimensions are more powerful than others in different relations. thus, we group relations with similar discriminant feature dimensions for parameter savings and better performance. in module 3, we train a binary classifier for each relation group so that it can predict triple's score in inference. the score is a soft prediction between 0 and 1, which indicates the probability of whether a certain triple exists or not. finally, we propose two novel negative sampling schemes, embedding-based and ontology-based, for classifier training in this work. they are used for hard negative mining, where these hard negatives cannot be correctly predicted by the baseline kge methods. we conduct extensive experiments and compare the performance and model sizes of greenkgc with several representative kgc methods on link prediction datasets. experimental results show that greenkgc can achieve good performance in low dimensions, i.e. 8, 16, 32 dimensions, compared with sota low-dimensional methods. in addition, greenkgc shows competitive or better performance compared to the high-dimensional kge methods with a much smaller model size. we also conduct experiments on a large-scale link prediction datasets with over 2.5m entities and show that greenkgc can perform well with much fewer model parameters. ablation studies are also conducted to show the effectiveness of each module in greenkgc. ## methodology greenkgc is presented in this section. it consists of three modules: representation learning, feature pruning, and decision learning, to obtain discriminant low-dimensional triple features and predict triple scores accurately. an overview of greenkgc is given in fig. 2 . details of each module will be elaborated below. ## conclusion and future work a lightweight kgc method, called greenkgc, was proposed in this work to make accurate link predictions in low dimensions. it consists of three modules that can be trained individually: 1) representation learning, 2) feature pruning, and 3) decision learning. experimental results in low dimensions demonstrate greenkgc can achieve satisfactory performance in as low as 8 dimensions. in addition, experiments on ogbl-wikikg2 show greenkgc can get competitive results with much fewer model parameters. furthermore, the ablation study shows the effectiveness of kg partitioning and feature pruning. modularized greenkgc allows several future extensions. first, greenkgc can be combined with new embedding models as initial features. in general, using a more expressive kge model can lead to better final performance. second, individual modules can be fine-tuned for different applications. for example, since the feature pruning module and the decision-learning module are supervised, they can be applied to various applications. finally, different negative sampling strategies can be investigated in different applications.