A New Pipeline for Knowledge Graph Reasoning Enhanced by Large Language Models Without Fine-Tuning (2024)

Abstract: Conventional Knowledge Graph Reasoning (KGR) models learn the embeddings of KG components over the structure of KGs, but their performances are limited when the KGs are severely incomplete. Recent LLM-enhanced KGR models input KG structural information into LLMs. However, they require fine-tuning on open-source LLMs and are not applicable to closed-source LLMs. Therefore, in this paper, to leverage the knowledge in LLMs without fine-tuning to assist and enhance conventional KGR models, we propose a new three-stage pipeline, including knowledge alignment, KG reasoning and entity reranking. Specifically, in the alignment stage, we propose three strategies to align the knowledge in LLMs to the KG schema by explicitly associating unconnected nodes with semantic relations. Based on the enriched KGs, we train structure-aware KGR models to integrate aligned knowledge to original knowledge existing in KGs. In the reranking stage, after obtaining the results of KGR models, we rerank the top-scored entities with LLMs to recall correct answers further. Experiments show our pipeline can enhance the KGR performance in both incomplete and general situations. Code and datasets are available.

## introduction knowledge graph (kg) is widely used to store enormous human knowledge or objective facts in the real world. conventional embedding-based kgr models learn structural embeddings for kg components. recently, path-based kgr models exploit the logical knowledge underlying the paths connecting the head and tail. all these models treat entities and relations as symbolized identifications without actual semantics and thus heavily rely on reasoning over the kg structures. however, even full-size kg datasets cannot fully cover the massive real-world knowledge and suffer from incompleteness, which naturally restricts kgr performances. although llms show exciting abilities, it is a challenge for them to singly act as entity reasoners for kgr task due to the huge kg entity space. @xcite further proves that matching the prediction of llms with entity names by postprocessing could easily fail. recently, kgt5 @xcite and csprom-kg @xcite have explored to learn kg structure by fine-tuning llms. however, on the one hand, for closed-source llms like chatgpt, we can not access the parameters and thus can not combine its knowledge with kgs by fine-tuning; on the other hand, fine-tuning open-source llms, such as llama3-70b 1 , for a single task is relatively expensive. therefore, how to assist kgr by incorporating the rich knowledge in llms and the structured information in kgs without fine-tuning becomes a remaining problem. relying on the instruction-following capability of llms, we propose to use llms from two views to enhance kgr performance without fine-tuning. first, many entity pairs in kgs lack necessary semantic relations because of the incompleteness of kgs. from the view of knowledge alignment, we align the knowledge in llms to the kg schema to mitigate the incompleteness of kgs before reasoning and then add the aligned knowledge into kgs in the form of edges, which preserves kg structures and enriches kg connections. formally, we input a pair of entities into llm and have it predict their relation. based on the enriched kgs, we can adopt arbitrary structure-aware kgr models to conduct the entity prediction task. second, after obtaining kg reasoning results, from the view of entity reranking, we leverage llms to rerank the top-scored entities of kgr models for further recalling the correct answers. finally, these two views of using llms to enhance kgr performance are not exclusive and together form our proposed three-stage pipeline for kgr: alignment, reasoning and reranking. moreover, in the alignment stage, we present three knowledge alignment strategies, including the closed domain strategy, the open domain strategy and the semi-closed domain strategy. they represent three kinds of approaches for inducing knowledge in llms to be outputted according to the kg schema. specifically, to directly align the knowledge to the manually predefined relations while constructing kgs, the closed domain strategy constrains llms to select one of the predefined relations in the form of multiple-choice questions. since the relations between entities in the real world go beyond the predefined ones, the open domain strategy does not restrict the output content, making less loss of information from llms. to provide explainable knowledge alignment for humans, in the semi-closed domain strategy, we map the output of llms in the open domain back to the predefined relations by semantic matching. to verify the effectiveness of our pipeline in incomplete and general situations, we conduct experiments on wn18rr and fb15k-237 with different sparse-level and full-size versions. additionally, we compare the accuracy and stability of the three alignment strategies to illustrate the quality of the generated relations. we further demonstrate the diverse influences of aligned edges on the original knowledge by analysing the llms output in the case study, which reveals that, when applying the open domain knowledge alignment, llms generate correct and fine-grained semantics beyond the predefined kg relations. this may explain the mechanism of performance enhancement. in summary, our contributions are tri-fold: • to solve the remaining challenges of llms in kgr, we propose a three-stage pipeline to assist and enhance conventional kgr models without fine-tuning: alignment, reasoning and reranking. ## methodology in this section, we describe the concrete implementation methodology of the new pipeline without fine-tuning. first, we propose three knowledge alignment strategies and the corresponding ways to convert the textual output of llms into kg schema. second, we train conventional structureaware kgr models over the enriched kgs. finally, we further leverage llms to rerank the top-scored entities of kgr models, recalling correct answers. ## conclusion this paper introduces a new pipeline for llms to assist and enhance kgr models without finetuning. we propose three knowledge alignment strategies to enrich kgs before reasoning and leverage llms as rerankers to recall correct answers. experiments illustrate the effectiveness of our pipeline, both in incomplete and general situations, and the accuracy and stability of the proposed knowledge alignment. the case study reveals the various outputs of llama3-70b and chatgpt.