KICGPT : Large Language Model with Knowledge in Context for Knowledge Graph Completion (2023)

Abstract: Knowledge Graph Completion (KGC) is crucial for addressing knowledge graph incompleteness and supporting downstream applications. Many models have been proposed for KGC and they can be categorized into two main classes, including triple-based and test-based approaches. Triple-based methods struggle with long-tail entities due to limited structural information and imbalanced distributions of entities. Text-based methods alleviate this issue but require costly training for language models and specific finetuning for knowledge graphs, which limits their efficiency. To alleviate the limitations in the two approaches, in this paper, we propose KICGPT, a framework that integrates a large language model (LLM) and a triple-based KGC retriever, to alleviate the long-tail problem without incurring additional training overhead. In the proposed KICGPT model, we propose an in-context learning strategy called Knowledge Prompt, which encodes structural knowledge into demonstrations to guide LLM. Empirical results on benchmark datasets demonstrate the effectiveness of the proposed KICGPT model with lighter training overhead and no finetuning.

## introduction knowledge graphs (kgs) are powerful representations of real-world knowledge. the relationships among entities are captured by triples of the @xmath0, relation, tail entity>. kgs serve as the foundation for various applications such as recommendation systems, question-answering, and knowledge discovery. knowledge graph completion (kgc) plays a crucial role for kgs by completing incomplete triples and hence addressing the inherent incompleteness of kgs. this paper focuses on the link prediction task in kgc, which is to predict the missing entity in an incomplete triple. based on the source of information used, existing kgc methods can be categorized into two * corresponding author main classes: triple-based methods and text-based methods @xcite . triple-based methods (e.g., transe @xcite , r-gcn @xcite , and hitter @xcite ) utilize the structure of the knowledge graph as the only source of information for kgc. given the typical imbalance of entities in kgs, long-tail entities are prevalent and the kgc task has limited structural information about them. consequently, due to information scarcity, the performance of triple-based methods tends to degrade when processing long-tail entities @xcite ). textbased methods (such as kg-bert @xcite ) mitigate this information scarcity problem by encoding textual description as an extra information source @xcite . besides the use of pre-trained language models (plms), finetuning remains necessary for text-based methods to handle diverse knowledge graphs. however, this is resource-intensive and requires task-specific finetuning for each downstream task. large language models (llms), such as chat-gpt and @xcite , have extensive internal knowledge repositories from their vast pretraining corpora, which can be used as an extra knowledge base to alleviate information scarcity for long-tail entities. @xcite , @xcite demonstrated the potential for llms to handle a wide range of tasks by speciallydesigned prompts without requiring training or finetuning. given the appealing properties of llms, a straightforward approach is to directly apply llms to kgc tasks. however, empirical evidence @xcite indicates that pure llms still cannot achieve state-of-the-art performance in kgc tasks such as link prediction. additionally, there are several challenges that hinder the application of llm on kgc tasks. first, the llm outputs can be unconstrained and may fall outside the scope of entities in the kgs. second, llms impose length limits on the input tokens, and the limits are far from sufficient for describing a complete kgc task. lastly, there is no effective in-context learning prompt design for llm on kgc tasks. to alleviate the aforementioned limitations, we propose in this paper a knowledge in context with gpt (kicgpt) framework. this integrates llms with a traditional structure-aware kg model (which is called a retriever). specifically, for each query @xmath1, r, ?) or @xmath2, r, t) in the link prediction task, where "?" denotes the missing tail or head entity to be predicted, the retriever first processes the query q independently and generates an ordered candidate entity list r retriever that ranks all entities in the kg based on their retrieval scores. the llm then performs re-ranking on the top m entities returned by r retriever , and replaces these m entities with re-ranked ones returned by the llm as the final result r kicgp t . to achieve the llm re-ranking, we propose knowledge prompt, a strategy of in-context learning (icl) @xcite , to encode the kg knowledge into demonstrations in prompts. note that in kicgpt we only need to train the retriever. compared with existing triple-based methods, kicgpt enables simultaneous utilization of knowledge sources (i.e., kg and llm's knowledge base) and facilitates their alignment and enrichment to alleviate information scarcity for long-tail entities. moreover, different from existing text-based models, kicgpt leverages a much larger semantic knowledge base without incurring additional training overhead. unlike directly applying the llm to the kgc task, which may generate undesired outputs, the proposed kicgpt constrains its output by formalizing link prediction as a re-ranking task for the given sequence r retriever . moreover, the standard link prediction task requires ranking for all the entities, which is not feasible for llm due to the length limit on input tokens. to overcome this limitation, kicgpt utilizes the retriever to obtain the top-m entities in r retriever , and only allows the llm to perform re-ranking on these entities. in summary, our contributions are as follows. triple-based kgc most existing kgc methods are triple-based methods, which complete the knowledge graph solely based on the triple information. early shallow knowledge graph embedding (kge) methods represent entities and relationships as low-dimensional embedding vectors in a continuous embedding space. based on the scoring function, these methods can be further categorized @xcite as translation-based (e.g., transe @xcite ) and semantic matching models (e.g., rescal @xcite and distmult @xcite ). however, they suffer from limited expressive power due to the use of shallow network structures. in recent years, more powerful network structures are integrated to solve kgc tasks. examples include the graph neural networks @xcite , convolutional neural networks @xcite , and transformer @xcite . most of these aggregate local structure context into node embeddings and achieve much improved performance. however, they are still limited by the imbalanced distribution of knowledge graph structure with insufficient knowledge about the long-tail entities. meta-learning @xcite and logical rules @xcite can mitigate the longtail problem in kg. the main difference between them and our work is that they handle long-tail entities by extracting and summarizing common structural patterns or rules from the limited information in kg, while kicgpt combines a vast external knowledge base inside the llm with the structural information in kgs, which can help alleviate information scarcity. text-based kgc in light of the success of natural language processing (nlp), text-based knowledge graph completion is gaining more attention. as another mode of knowledge different from the structured kg, text can provide rich semantic information. dkrl @xcite first introduces textual descriptions into entity embeddings produced by a convolutional neural network. subsequent works (such as kg-bert @xcite , kepler @xcite , and pretrain-kge @xcite ) use a pre-trained language model (plm) to encode the text descriptions. more recently, lmke @xcite proposed a contrastive learning framework that uses the plm to obtain entity and relation embeddings in the same space as word tokens, and demonstrated its effectiveness on long-tail problem these methods generally rely on the language models to process text descriptions and require finetuning for different knowledge graphs. the proposed kicgpt, which uses llm directly, is more efficient because it is training-free and requires no finetuning. llms for kgs some recent works also explore the ability of llm on kg tasks. structgpt @xcite proposed a general framework for improving zero-shot reasoning ability of llms on structured data. it leverages the llm to perform reasoning on the kg question answering (kgqa) task with the help of auxiliary interfaces that fetch the needed information from kg. though both structgpt and our work utilize kg and llm, structgpt aims to help the llm to handle structured data and explore kgqa tasks, while ours utilizes the knowledge base inside the llm to handle kgc's long-tail problem. moreover, structgpt performs multi-step reasoning directly based on the kg structure, while the proposed kicgpt utilizes the kg information in a different way. first, we use the whole kg to generate preliminary results before using the llm. moreover, we incorporate a portion of the kg triples into icl demonstrations to help the llm conduct reasoning. @xcite directly evaluated the performance of a llm on kg reasoning. our work design an icl strategy to guide the llm to perform reasoning. moreover, their experiments on the link prediction task only involve 25 sampled instances from the fb15k-237 dataset (which has 20,466 test triples). instead, we re-run their setting on chatgpt and report the new results as a baseline in our experiments. icl for llms a typical way of approaching llm is through in-context learning @xcite , by providing explicit instructions and demonstrations to guide the model's behavior. this approach has been effective in various language understanding and generation tasks @xcite . in-context learning exposes the model to specific contextual information, allowing it to grasp and reproduce necessary patterns and structures for precise generation @xcite . however, the success of in-context learning depends heavily on the quality of the prompt, and crafting suitable prompts can be delicate @xcite . while there has been work exploring icl on different tasks @xcite , to the best of our knowledge, no such work has been done for kgc. unlike existing works, the proposed icl strategy, knowledge prompt, considers the characteristics of kgc tasks and demonstrates its effectiveness on kgc tasks. ## methodology in this section, we introduce the proposed kicgpt model. the complete algorithm is in appendix a.1. ## experiment in this section, we empirically evaluate kicgpt.