LLM as Prompter: Low-resource Inductive Reasoning on Arbitrary Knowledge Graphs (2024)

Abstract: Knowledge Graph (KG) inductive reasoning, which aims to infer missing facts from new KGs that are not seen during training, has been widely adopted in various applications. One critical challenge of KG inductive reasoning is handling low-resource scenarios with scarcity in both textual and structural aspects. In this paper, we attempt to address this challenge with Large Language Models (LLMs). Particularly, we utilize the state-of-the-art LLMs to generate a graph-structural prompt to enhance the pre-trained Graph Neural Networks (GNNs), which brings us new methodological insights into the KG inductive reasoning methods, as well as high generalizability in practice. On the methodological side, we introduce a novel pretraining and prompting framework ProLINK, designed for low-resource inductive reasoning across arbitrary KGs without requiring additional training. On the practical side, we experimentally evaluate our approach on 36 low-resource KG datasets and find that ProLINK outperforms previous methods in three-shot, one-shot, and zero-shot reasoning tasks, exhibiting average performance improvements by 20%, 45%, and 147%, respectively. Furthermore, ProLINK demonstrates strong robustness for various LLM promptings as well as full-shot scenarios.

## introduction knowledge graph (kg) reasoning, also known as kg link prediction, aims at inferring new facts from existing kgs in the triple format (head entity, relation, tail entity) @xcite . this technique has been widely studied and applied in various domains, including information retrieval, e-commerce recommendations, drug discovery, and financial prediction @xcite * the corresponding author figure 1 : illustrations of knowledge graphs and corresponding relation graphs. the interaction edge h2t means the source relation has a head entity in the kg which is a tail entity of the target relation. @xcite @xcite . for example, the query (q 1 ) in figure 1 consists of 'entity 3' and a relation type 'occupation'. its expected answer for kg reasoning is 'entity 6'. the dynamic nature of real-world kgs fosters recent research interest in inductive kg reasoning-inferring from new kgs which have entities/relations unseen during training @xcite . in contrast to earlier methods that learn graph-specific embeddings for the training kg @xcite @xcite , recent inductive reasoning models are trained to address the query (q 1 ) in the training kg, enabling them to answer the query (q 2 ) in a fully new kg @xcite . as illustrated in figure 1 , this transferability stems from the shared interaction patterns in relation graphs, which can be captured by graph neural networks (gnns) @xcite @xcite without using entity/relation textual information. one critical challenge of kg inductive reasoning is handling low-resource scenarios marked by scarcity of both textual and structural information. for the query (q 3 ) in figure 1 , the inference kg lacks textual context for entities and has no support triple for the 'co-worked' relation. such lowresource scenarios frequently occur in specialized or emerging domains due to long-tail distribution and new relation additions @xcite , hindering the wide adoption of kg inductive reasoning. text-based methods, even the current powerful large language models (llms) @xcite , are constrained by limited textual data and complicated graph structures, while graph-based methods struggle with few-shot relation types that lack enough interaction edges in the relation graph 1 . human reasoning, however, may address the above low-resource queries without expert knowledge or prior learning. simply leveraging limited relation semantics to understand graph structures, humans can successfully deduce that the 'coworked' relation should connect two persons, suggesting the answer entity can be a head entity of the 'job' relation and near the query entity 'entity 4'. inspired by the human reasoning process which mostly relies on relation semantics, we pose a question: can llms be used to emulate human reasoning for relation semantic understanding, thereby enhancing the gnn-based inductive reasoning? this work gives a positive answer to the above question and presents an approach that leverages llm's basic language power to improve kg inductive reasoning in any low-resource scenarios without additional model training for new kgs. this ability is crucial for elevating the generalizability of ai technologies to handle data dynamics in the real world. specifically, we propose a novel pretraining and prompting framework prolink for low-resource inductive reasoning across arbitrary kgs. first, prolink pretrains a gnnbased kg reasoner with novel techniques enhancing few-shot prediction performance. then, given a new inference kg with sparse relation types, pro-link employs a pre-trained llm to construct a prompt graph through concise relation descriptions (a dozen words per relation). the prompt graph is calibrated to eliminate noise and subsequently injected into the relation graph of the kg to improve the performance of the gnn reasoner. in summary, this work has four novel contributions: • we introduce a new kg reasoning problem, i.e., low-resource inductive reasoning on kgs with arbitrary entity and relation vocabularies, which generalizes most kg link prediction tasks. in this work, we focus on the low-resource challenge of kg inductive reasoning, inferring not merely unseen but also few-shot relation types. existing methods for few-shot link prediction @xcite @xcite either only work on the training graph structure or have to use graph-specific textual or ontological information to train, therefore cannot achieve inductive reasoning on arbitrary kgs. other related studies also have difficulty accom-plishing this low-resource task, including entitylevel inductive reasoning @xcite @xcite ) and text-based inductive reasoning @xcite @xcite . detailed related work is introduced in the appendix e. ## methodology this work presents a novel pre-training and prompting framework, prolink, which employs pretrained language models (llms) and graph neural networks (gnns) to handle semantic and structural information, respectively. as illustrated in figure 3 , for a few-shot query relation in a new inferring kg, the pre-trained gnn reasoner (section 3.1) infers from kg subgraphs without model finetuning, guided by a relation-specific prompt graph. the prompt graph is constructed by a frozen llm prompter (section 3.2) from relation semantics, and then calibrated by prompt calibrator (section 3.3) to mitigate noise. ## experiments we extensively evaluate our method on k-shot inductive reasoning of kgs. in particular, we wish to answer the following research questions: rq1: how effective is our method under the low-resource scenarios of distinct kgs? rq2: how do different llm prompts impact prompt graphs and kg inductive reasoning? rq3: how do the main components of our method impact the performance? rq4: how does our method perform in more shots and full-shot scenarios? rq5: how efficient is our model compared with traditional approaches? rq6: how does gnn reasoning change before and after injecting the prompt graph in case studies? due to the space limitation, discussions about rq5 and rq6 are detailed in the appendix. ## conclusions we propose a novel pre-training and prompting framework, prolink, for low-resource inductive reasoning across arbitrary kgs. to generate an effective prompt graph for few-shot relation types, we design enhancing techniques for the gnn reasoner and instruction prompts for the llm prompter. besides, a novel prompt calibrator is proposed to mitigate the potential noise and achieve the information alignment of the above two components. extensive experiments have verified that the prolink achieves significant performance in both few-shot and zero-shot scenarios. besides, prolink requires no model fine-tuning, thereby having advantages of better efficiency and scalability than previous gnn-based methods. here, we discuss two potential limitations of pro-link. first, unlike recent gnn-based fully inductive methods, our method requires a brief relational context for each relation type. despite our efforts to minimize text requirements, it may be unavailable in certain scenarios without manual inputs from users. second, the usage of textual context in our method is not sufficient enough. to circumvent the need for additional model training, we streamline the llm queries by only asking for entity types. incorporating semantic embeddings into gnns or fine-tuning the llm prompter to leverage relation semantics could further enhance performance. exploring this avenue will constitute a primary focus of our future work. the potential risks of our work may include generating factual triples about privacy or fake information, depending on the legality and reliability of the input data. in addition, we utilize the chatgpt ai assistant when polishing some paragraphs of the paper draft.