KB -Plugin: A Plug-and-play Framework for Large Language Models to Induce Programs over Low-resourced Knowledge Bases (2024)

Abstract: Program induction (PI) has become a promising paradigm for using knowledge bases (KBs) to help large language models (LLMs) answer complex knowledge-intensive questions. Nonetheless, PI typically relies on a large number of parallel question-program pairs to make the LLM aware of the schema of a given KB, and is thus challenging for many low-resourced KBs that lack annotated data. To this end, we propose KB-Plugin, a plug-and-play framework that enables LLMs to induce programs over any low-resourced KB. Firstly, KB-Plugin adopts self-supervised learning to encode the detailed schema information of a given KB into a pluggable module, namely schema plugin. Secondly, KB-Plugin utilizes abundant annotated data from a rich-resourced KB to train another pluggable module, namely PI plugin, which can help the LLM extract question-relevant schema information from the schema plugin of any KB and utilize the information to induce programs over this KB. Experiments show that KB-Plugin outperforms SoTA low-resourced PI methods with 25x smaller backbone LLM on both large-scale and domain-specific KBs, and even approaches the performance of supervised methods.

## introduction recently, the usage of knowledge bases (kbs) as external resources to assist large language models (llms) @xcite in answering complex knowledge-intensive questions has gained increasing study @xcite @xcite . among various methods, program induction (pi) has emerged as a promising paradigm due to its good interpretability and capacity to support complex reasoning operations @xcite @xcite . given a kb, pi methods employ llms to convert a question into a multi-step program (e.g., kopl @xcite and s-expression @xcite ), whose execution against the kb produces the answer. despite strong capacity, most pi methods rely on individual training for each kb using a large number of manually annotated questionprogram pairs @xcite @xcite . as for many low-resourced kbs that lack program annotations, how to enable llms to utilize their knowledge via pi remains a challenging problem. recent studies @xcite have indicated that the mapping from questions to program sketches (i.e., composed functions without arguments, such as @xmath0. hence the main challenge for pi over low-resourced kbs is to determine the argument for each function @xcite , which requires llms to link natural language in a question to corresponding schema items (i.e., pre-defined relations and concepts) in the kb (e.g., in fig 1 , the relation "part of network" and the concept "rail network" are arguments of function relate and filterconcept, respectively), so it is important to provide llms adequate information of each schema item. a straightforward approach is to directly feed all the schema information to the llm via a prompt. however, the broad schema of kbs and limited context windows of llms make this infeasible @xcite . regarding the above challenges, we are inspired by recent studies that claim the parameters of llms can encode task-specific knowledge @xcite @xcite . our basic idea is to encode detailed schema information of a kb into the parameters of a pluggable module (e.g., lora @xcite ), namely schema plugin, so as not to be hampered by limited context windows like the prompt-based approach. then we use another pluggable module, namely pi plugin, to help the llm capture question-relevant schema information from the schema plugin and utilize this information to induce programs. as illustrated in fig. 1 , by simply plugging the schema plugin of a kb and the pi plugin, the llm is injected with the schema information of this kb and the ability to induce programs over it. we name this framework kb-plugin. to implement kb-plugin, there remain two key problems: (1) by what task can sufficient information about each schema item in a kb be encoded into its schema plugin? (2) without annotated data from the low-resource kbs, how can the pi plugin learn to extract and utilize questionrelevant schema information from their schema plugins to induce programs over these kbs? to solve the above problems, we propose a novel plugin learning and transfer framework. first, inspired by prior studies @xcite which show that schema items in a kb can be well represented by fact triples involving them, we propose to learn schema plugins via a self-supervised triple completion task. specifically, given a kb, we plug a schema plugin into the llm and tune the plugin to enable the llm to complete relevant triples for each schema item in the kb. in this way, the detailed schema information can be encoded into this schema plugin. as for pi plugin learning, inspired by @xcite , we utilize abundant program annotations from a rich-resourced kb. specifically, we use this kb to generate multiple kbs with different schemas via alias replacement and train a schema plugin for each of them. given a training question, we plug these schema plugins alone with the pi plugin into the llm in turn and train the pi plugin to make the llm generate the correct program whose arguments conform to the currently plugged schema plugin. in this way, the pi plugin is forced to learn the skills of extracting and utilizing question-relevant schema information from the plugged schema plugin for pi over the corresponding kb. besides, since the pi plugin is trained to be compatible with different schema plugins, it can be directly transferred to other low-resourced kbs and generalize well with their schema plugins, even if most schema items in these kbs are unseen during its training. in experiments, we take wikidata-based kqa pro as the rich-resourced kb to train the pi plugin, and evaluate our framework on three largescale freebase-based datasets (webqsp, graphq, and grailqa) and two domain-specific datasets (metaqa for movie domain and soaybench for academic domain). the results show that kb-plugin outperforms sota low-resourced pi methods with 25× smaller backbone llm, demonstrating its scalability to extremely complex schemas and adaptability to various domains. on graphq, grailqa, and metaqa, kb-plugin even surpasses the performance of several supervised methods. our contributions include: (1) proposing kb-plugin, a novel plug-and-play framework that enables llms to induce programs over any lowresourced kb; (2) empirically validating the efficacy of kb-plugin for both large-scale and domainspecific kbs through extensive experiments. ## problem formulation in this section, we first provide some necessary definitions and then formulate our task. knowledge base. a knowledge base (kb) can be formalized as @xmath1, e, r, t }, where c, e, r and t represent the sets of concepts, entities, relations and fact triples, respectively. specifically, @xmath2, r c } ∪ r l , where r e is "instance of", r c is "subclass of", and r l is the set of other general relations. correspondingly, t can be divided into there disjoint subsets: (1) "instance of" triples t @xmath3, r e , c)|@xmath4, @xmath5; (2) "subclass of" triples t @xmath6, r c , c j )|c i , c @xmath7; (3) relational triples t @xmath8, r, e j )|e i , e @xmath9, @xmath10. elements in c and r are also called the schema items of kb. program induction. given a kb kb and a natural language question @xmath11, w 2 , • • • , w |x| , program induction (pi) aims to convert x into a program y, which would return the correct answer when executed against kb. formally, y is composed of functions that take a specific type of arguments, and can be serialized as @xmath12, • • • , f t (arg t ), • • • , f |y| (arg |y| ) , f @xmath13, arg @xmath14. here, f is a set of pre-defined functions that cover basic reasoning operations on kbs. in this work, we use kopl @xcite as our programming language. task formulation. suppose we have access to ( 1 ) source kb kb s and source domain data d @xmath15, y s i )} n s @xmath16, which are question-program pairs for kb s ; (2) target kb kb t , which is lowresourced and has no annotated data. the goal is to learn a pi model m t p i that can translate a question x t for kb t into program y t , whose execution on kb t produces the correct answer. ## methodology as mentioned in the introduction, to enable a llm m to induce programs over low-resourced kb t , kb-plugin learns two types of pluggable modules for m : (1) kb-specific schema plugin m sc , which stores information of schema items of a given kb within its parameters; (2) kb-transferable pi plugin m p i , which encodes the skill of inducing programs over any kb by extracting and utilizing question-relevant schema information from the schema plugin of this kb. it is trained with kb s and d s but can be directly transferred to kb t . the final pi model for kb t can be formulated as ## conclusion we propose kb-plugin, a plug-and-play framework that enables llms to induce programs over any low-resourced kbs by learning two types of pluggable modules: kb-specific schema plugin and kb-transferable pi plugin. kb-plugin achieves better or comparable performance on five heterogeneous kbqa datasets with much smaller backbone llms compared to sota pi methods for low-resourced kbs, demonstrating its effectiveness for both large-scale and domain-specific kbs. ablation study and case study also prove the rationality and further showcase the advantage of kb-plugin.