Path-enhanced Pre-trained Language Model for Knowledge Graph Completion (2025)

Abstract: Pre-trained language models (PLMs) have achieved remarkable knowledge graph completion(KGC) success. However, most methods derive KGC results mainly from triple-level and text-described learning, which lack the capability to capture long-term relational and structural information. Moreover, the absence of a visible reasoning process leads to poor interpretability and credibility of the completions. In this paper, we propose a path-enhanced pre-trained language model-based knowledge graph completion method (PEKGC), which employs multi-view generation to infer missing facts in triple-level and path-level simultaneously to address lacking long-term relational information and interpretability issues. Furthermore, a neighbor selector module is proposed to filter neighbor triples to provide the adjacent structural information. Besides, we propose a fact-level re-evaluation and a heuristic fusion ranking strategy for candidate answers to fuse multi-view predictions. Extensive experiments on the benchmark datasets demonstrate that our model significantly improves the performance of the KGC task.

## introduction knowledge graphs (kgs) are designed to store knowledge in graph-structured format, as seen in freebase @xcite , wordnet @xcite , and nell @xcite . the extensive application of various kgs has greatly benefited numerous downstream tasks, such as question answering, recommendation systems, and information retrieval. however, due to the limited scale of kgs, whether manually or automatically constructed, they invariably suffer from incomplete coverage and fail to encompass the vast expanse of real-world knowledge. this limitation has given rise to the task of knowledge graph completion (kgc), which involves predicting missing links by understanding the existing triples within kgs. typically, most mainstream kgc methods fall into two categories: embedding-based and pathbased. embedding-based methods @xcite @xcite focus on mapping entities and relations into a low-dimensional, continuous vector space to capture intrinsic connections and predict missing links in the vector space. by comparison, path-based methods @xcite @xcite aim to use paths between entities to predict and achieve more directly interpretable results. recently, inspired by the success of pre-training language models (plms), such as bert @xcite , roberta @xcite , and t5 @xcite , in nlp tasks, encouraging increased interest in probing plms to complete kgs. according to the structure of the models, plm-based models can be divided into two categories: encoder-only models, such as kg-bert @xcite and pkgc @xcite , and encoder-decoder models, such as kgt5 @xcite and kgs2s @xcite . the encoder-only models encode both the query and all candidate entities to calculate their matching confidence, while the encoder-decoder models encode the query and then decode possible candidate entities. overall, plm-based kgc meth-ods work by converting triples into serialized text descriptions sentences and feeding the sentences to plms to complete kgs. as illustrated in figure 1 , current plm-based models (referred to as triple-level plm) encode the concatenated query text, "tu youyou nationality [mask]", from a "flat" text-described view and directly generate the answer entity, china. however, current plm-based models have flattened triples into a simple triple-level text description presenting three significant limitations: (i) low expressiveness. the embeddings of entities and relations are learned at the triple level, which represents the embeddings as being focused only on a local perspective (i.e., one-hop structure). previous research has indicated that relying solely on local relational information for kg learning is not enough @xcite . (ii) inefficient information propagation. these models depend exclusively on one-hop neighbors for aggregating and propagating information, which is inefficient for transferring semantics and knowledge between entities. (iii) lack of interpretability. kgc results are generated directly in response to queries without the necessary explanations or reasoning process, which harms the credibility of completions. therefore, there is an urgent need to leverage rich structural information of kgs to capture more complex, long-term, or higher-level features to enhance expressiveness, propagation, and interpretability. to address the limitations above, in this paper, we propose a path-enhanced pre-trained language model-based knowledge graph completion model (pekgc), which employs multi-view generation by an encoder-decoder architecture to tackle lacking long-term relational structures and poor interpretability issues. to enable plm-based models to capture path-level knowledge and bridge longterm gaps between entities, our model generates possible entities with reasoning path chains, which also serve as evidence for the generated answers in response to queries. besides, to extract local knowledge, our model simultaneously generates answers at a triple level. in this way, the model can focus on multi-level knowledge simultaneously and can be cross-validated occurring at both levels. to better use the adjacent structural information and enhance the inference ability of the model, we also design a neighbor selector module, which is pre-trained to filter the most relevant triples to the query as neighborhood information. in addition, we use kg soft prompts and position soft prompts to distinguish between kg knowledge and textual knowledge, as well as to mark the internal relationships of triples. to further re-evaluate and re-rank the joint results of both levels, we propose a fact-level re-evaluation and a heuristic fusion ranking strategy, which can re-calculate the confidence scores of candidate answers and efficiently re-rank them. in summary, the major contributions of this work are as follows: ## experiments to investigate our model's effectiveness and efficiency, we evaluate the performance of proposed model on kgc (link prediction) task, which aims to produce a ranking list of all entities for a query ((e h , r, ?) or (?, r, e t )), on benchmark kgs. we also conduct an ablation study to demonstrate the impact of each proposed module. additionally, we present examples of reasoning paths in a case study to illustrate that pekgc effectively generates highquality reasoning paths. ## conclusion we propose a multi-view generation framework, pekgc, for kgc tasks that captures triple-level knowledge, long-term path-level knowledge, and structural neighborhoods, achieving more directly interpretable results. we propose a heuristic fusion ranking strategy for multi-view generation during inference to further combine triple-level and pathlevel results to achieve better ranking performance. experimental results demonstrate the effectiveness of our approach. in future work, we aim to address the challenges of diversity and generalization of plm-based models.