Joint Pre-Encoding Representation and Structure Embedding for Efficient and Low-Resource Knowledge Graph Completion (2024)

Abstract: Knowledge graph completion (KGC) aims to infer missing or incomplete parts in knowledge graph. The existing models are generally divided into structure-based and description-based models, among description-based models often require longer training and inference times as well as increased memory usage. In this paper, we propose Pre-Encoded Masked Language Model (PEMLM) to efficiently solve KGC problem. By encoding textual descriptions into semantic representations before training, the necessary resources are significantly reduced. Furthermore, we introduce a straightforward but effective fusion framework to integrate structural embedding with pre-encoded semantic description, which enhances the model’s prediction performance on 1-N relations. The experimental results demonstrate that our proposed strategy attains state-of-the-art performance on the WN18RR (MRR+5.4% and Hits@1+6.4%) and UMLS datasets. Compared to existing models, we have increased inference speed by 30x and reduced training memory by approximately 60%.

## introduction knowledge graph contains of a series of triples, depicting relations between entities in the form of graphs or triples. these entities can encompass real world entities such as objects, concepts, events, etc., while relations denote their connections and interactions. the composition format of a triple is (h,r,t), where h and t are head entity and tail entity, and r represents the relation connecting the two entities. popular knowledge graphs include freebase @xcite , worldnet @xcite , and wikidata @xcite . although these knowledge graphs are composed of massive amounts of real-world data, many parts are still missing. kgc aims to complete knowledge graph information by accurately predicting unknown relations through leveraging known entities and relations. a comprehensive knowledge graph can be further used for downstream tasks, such as question answer @xcite , relation extraction @xcite , recommendation systems @xcite , etc. existing kgc methods can be divided into two categories: embedding-based model and description-based model. embedding-based models aiming to reflect real triples as faithfully as possible in the corresponding real or complex vector space through learning embedding representations, modeling entities and relations by capturing the inherent structure of the graph. description-based models use textual descriptions of entities and relations as information to predict missing entities. most description-based models using text directly as input, the training requires substantial memory and time due to the necessity of processing sufficiently long sequences, the inference speed may significantly increase as well. for example, kg-bert requires a large amount of gpu memory and several days for inference over the entire knowledge graph, making training extremely challenging under limited resources. the primary cause of these issues is the substantial volume of text inputs, which aggravates the processing pressure on the model. in order to reduce the length of input, we use the pooling features from the last layer instead of the entire description text, as we assume that the pooling features can effectively represent the overall semantic information of description. therefore, pre-encoding all descriptions with a pre-trained language model can effectively reduce resource consumption. besides, recently studies have shown that combining structural embedding and description representation can improve model performance. this type of method incorporates structural information into the description-based model. however, some joint-based models inappropriately employ descriptive features for structural processing, which may lead to the disruption of description semantic. the main contributions of this paper are as follows: the structure of this article is as follows. we first introduce the related work of the kgc algorithm in § 2. details about our model and fusion framework will be reported in § 3. the experimental results and further analysis will be explained in § 4 and § 5, respectively. finally, we summarize our research in § 6. ## related work knowledge graph completion algorithm aims to learn knowledge from existing knowledge graph and predicts missing entities to complete it. transe @xcite models triples as translation paradigm h + r ≈ t. transh @xcite projects each relation onto a hyperplane through a matrix. transr @xcite establishes distinct entity and relation vector space to enhance modeling capabilities. rotate @xcite projects entities and relations into complex space, defining relations as rotational transforma-tions from head entities to tail entities. distmult @xcite restricts the relation matrix to be a diagonal matrix, greatly reducing the number of parameters but making it challenging to model asymmetric relations. complex @xcite addresses complex relation modeling by incorporating complex space. quate @xcite introduces hyper-complex space on top of complex. with the advancement of deep learning, convolutional neural network models @xcite and attention-based models @xcite have also been introduced for link prediction tasks. the description-based model uses textual descriptions of entities and relations for link prediction. ssp @xcite simultaneously learns topic models and em-bedding models to enhance the correlation between descriptions and triples. kg-bert @xcite applies bert to encode context in description for knowledge graph completion. mtl-kgc @xcite and lp-bert @xcite extends language model by introducing a multi-task framework for kgc. mlmlm @xcite ) employ masked language model to encode missing entities. ke-pler @xcite ) constructs a unified pre-trained language model and knowledge graph embedding model, handling both natural language tasks and kgc tasks. kgt5 @xcite developed a link prediction method using the sequence to sequence framework. ghn @xcite innovatively uses hard negatives mining and contrastive learning framework for kgc. in addition, there has been an increasing research emphasis on joint-based models that integrating structure embedding and description representation in recent years. pretrain-kge @xcite replaces embedding vectors in embedding models with description representations processed by bert. blp @xcite encode node descriptions in order to train relation vectors for embedding models. star @xcite introduces a siamese-style hybrid framework and integrates the results of rotate. lass @xcite fine-tunes bert with the loss of embedding models. kglm @xcite embeds structures into pre-trained language models and utilizes the language model to learn graph representations. jointse @xcite introduces an enhancement module on top of fusion to filter out irrelevant information from the text. kicgpt @xcite uses structural knowledge as a knowledge prompt to guide llm learning. ## method in this section, we introduce pemlm and its fusion version, pemlm-f. we first introduce the problem formulation in §3.1. the model structure of pemlm is proposed in §3.2, with the supplement of the fusion framework in §3.3. ## experience in §4.1, we will provide an overview of the experiment detailed setup. in §4.2 we will focus on the specific results and analysis. ## discussion in this section, we will investigate four questions: how to pre-encode a good description representation ( §5.1), how to properly fuse structure embeddings with description representations ( §5.2), why we describe pemlm as efficient and low-resource ( §5.3), and the performance of pemlm on different type of relations ( §5.4). ## conclusion in this paper, we introduce a pre-encoded masking language model for efficient knowledge graph completion. by pre-encoding semantic representations extracted from description texts, we significantly reduce the memory requirement and inference time. to integrate structural and semantic information effectively, we build a learnable fusion module that integrates the information of both representations. experiment results demonstrate that our pemlm-f surpasses the majority of baseline metrics on the wn18rr dataset. overall, the model achieves state-of-the-art performance on both umls and wn18rr datasets, while significantly enhancing inference speed and reducing memory consumption.