Few-shot In-context Learning on Knowledge Base Question Answering (2023)

Abstract: Question answering over knowledge bases is considered a difficult problem due to the challenge of generalizing to a wide variety of possible natural language questions. Additionally, the heterogeneity of knowledge base schema items between different knowledge bases often necessitates specialized training for different knowledge base question-answering (KBQA) datasets. To handle questions over diverse KBQA datasets with a unified training-free framework, we propose KB-BINDER, which for the first time enables few-shot in-context learning over KBQA tasks. Firstly, KB-BINDER leverages large language models like Codex to generate logical forms as the draft for a specific question by imitating a few demonstrations. Secondly, KB-BINDER grounds on the knowledge base to bind the generated draft to an executable one with BM25 score matching. The experimental results on four public heterogeneous KBQA datasets show that KB-BINDER can achieve a strong performance with only a few in-context demonstrations. Especially on GraphQA and 3-hop MetaQA, KB-BINDER can even outperform the state-of-the-art trained models. On GrailQA and WebQSP, our model is also on par with other fully-trained models. We believe KB-BINDER can serve as an important baseline for future research. We plan to release all the code and data. Our code is available at https://github.com/ltl3A87/KB-BINDER .

## introduction question answering over knowledge bases (kbqa) @xcite has been a long-standing research problem in the ai community. it has attracted wide attention from the community with its significant role in making large-scale knowledge bases accessible to non-expert users @xcite @xcite . however, despite the fact that the increasing scale of knowledge bases can enable the retrieval with higher coverage on miscellaneous topics, it poses a great challenge for suppliers with limited resources, who rely on the models trained on certain knowledge bases or benchmarks. concretely, the difficulties primarily lie in the following aspects: 1) data intensiveness: larger knowledge bases require ever larger quantities of annotated data to allow fine-tuned models to generalize well over them. @xcite @xcite . 2) dataset specificity: for relatively small-scale kbqa datasets, the fully-trained models tend to overfit to a specific schema, and can hardly generalize to knowledge base questions in unseen domains @xcite @xcite . these challenges make it crucial to devise a new framework that can work in both low-resource and training-free settings in kbqa. recently, large language models (llms) like @xcite have demonstrated their strong generalizability @xcite @xcite @xcite on a wide range of text, table, commonsense and even math qa tasks with fewshot in-context learning. other works also validate that codex @xcite can parse and transform unstructured instructions to structured and executable code with only a few dozen demonstrations @xcite . these works inspire us to tackle kbqa with llms, an under-explored area in the literature that is particularly challenging compared to other qa tasks because of the massive scale of modern kbs. however, it is still unclear how to address kbqa with in-context learning. unlike many other question-answering tasks, where the evidence is provided with a reasonable length limit, kbqa needs to condition on a massive graph containing millions of nodes and billions of edges. evidently, it is impossible to feed the whole graph as-is to the language model. even feeding a subgraph is extremely challenging as it requires splitting the monolithic graph into self-consistent and queryrelevant chunks, which is itself an unaddressed research problem. without feeding the knowledge graph as an additional input, language models become unaware of the schema of the kb. this problem makes it difficult to associate surface forms in the questions with the corresponding entities and relation types in a specific kb, not to mention generate executable logical forms with these linked entities and relations. these challenges make it hard to build in-context kbqa systems. in this work, we propose kb-binder, which, for the first time, enables training-free few-shot in-context learning on kbqa. our framework consists of two stages as shown in figure 1 . in the first stage, we demonstrate a few kbqa questions and their corresponding logical forms as the exemplary pairs for codex to generate a draft of an unseen question. the draft is a 'preliminary' logical form likely to contain mistakes in both entities and relations. for example, due to a lack of information about the kb schema, codex might generate a draft containing 'medicine.manufactured_drug.shape' while the true relation in the kb should be 'medicine.manufactured_drug_form.shape'. in the second stage, kb-binder binds the 'preliminary' entities to the true entity by using a lexicon-based similarity search over the whole kb. once the entities are bound, we search through the vicinity of the bound entities to bind the 'preliminary' relations. we fill the bound entities and relations into the draft to generate a set of 'refined' logical forms. we execute these logical forms against the kb and return the executed results as the answer. to enhance kb-binder with more pertinent exemplars, we also propose a kb-binder-r with retrieved exemplars from the training set. in general, previous works rely heavily on predefined heuristics for a target knowledge base to find the potential candidates @xcite @xcite . kb-binder, however, does not need heuristics customized to specific kb schema due to the inherent generalizability of llms. we test the performance of our models under few-shot setting on four public datasets, we-bqsp @xcite , grailqa @xcite , graphqa @xcite and metaqa @xcite . on graphqa and 3-hop metaqa, kb-binder achieves 39.5 f1 and 99.5% hits@1 scores respectively, surpassing the previous sota by 7.7 on f1 score and 0.6% on hits@1 correspondingly. on webqsp, kb-binder-r can achieve 74.4% f1 score, only 4.4% lower than the sota model @xcite . these experimental results demonstrate the effectiveness of our approach. given the simplicity and generality of kb-binder, we believe it could serve as an important baseline for future kb research, especially in the low-resource setting. in-context learning with llms. in-context learning with large language models @xcite has shown strong few-shot performance in many nlp tasks, such as question answering @xcite , information extraction @xcite , and numerical reasoning @xcite . analyses into the mechanisms behind this behavior are undertaken by @xcite ; @xcite . empirically, @xcite shows the effectiveness of constructing prompts using an input-label pairing format, and @xcite experiment with the number of examples provided, as well the idea of retrieving relevant examples to a test input to construct the prompt with. these results inform the prompt-construction methods used in our work. @xcite suggests that incorporating explanatory task instructions in context can improve performance, however, we leave a deeper exploration of this to future works. ## methodology given a new question, kb-binder leverages an llm to generate a preliminary logical form as a draft. a draft is not guaranteed to be executable, as it is generated by the llm without being explicitly restricted to the candidates' vocabulary and knowledge graph structure. however, with the demonstration of in-context prompting, drafts can reveal the structural relationships among mentioned entities in a semantically reasonable way. as a result, the generated drafts can simplify the search space needed to retrieve real entities and schema terms. these entities and terms are then used to revise the draft to a real executable logical form for a given question. this process is illustrated in figure 2 . ## experiment in this section, we briefly introduce the benchmarks used to evaluate the performance of our framework. and we demonstrate the detailed setting of kb-binder and its result on each of the datasets compared with the fully-trained baselines. ultimately, we make an analysis of the variation of design choices and their corresponding potential causes. ## conclusion kb-binder is the first framework that enables the challenging few-shot learning on kbqa with the reasoning capability of large language models. it first generates drafts with llm as preliminary logical forms, and then binds the entities and schema items of the drafts to the target knowledge base iteratively until an executable one can be found. kb-binder (k) adopts majority voting, further enlarging the proportion of answerable questions with the help of more diverse formats of top k drafts. kb-binder (k)-r with retrieved exemplars is proved to be especially advantageous when applied to i.i.d questions. in general, kb-binder and its derivatives achieve strong performance on all the common-used kbqa datasets we select, and we hope it can set a strong baseline for future work on kbqa with a low-resource setting.