Few-shot Transfer Learning for Knowledge Base Question Answering: Fusing Supervised Models with In-Context Learning (2024)

Abstract: Existing Knowledge Base Question Answering (KBQA) architectures are hungry for annotated data, which make them costly and time-consuming to deploy. We introduce the problem of few-shot transfer learning for KBQA, where the target domain offers only a few labeled examples, but a large labeled training dataset is available in a source domain. We propose a novel KBQA architecture called FuSIC-KBQA that performs KB-retrieval using multiple source-trained retrievers, re-ranks using an LLM and uses this as input for LLM few-shot in-context learning to generate logical forms, which are further refined using execution-guided feedback. Experiments over four source-target KBQA pairs of varying complexity show that FuSIC-KBQA significantly outperforms adaptations of SoTA KBQA models for this setting. Additional experiments in the in-domain setting show that FuSIC-KBQA also outperforms SoTA KBQA models when training data is limited.

## introduction a knowledge-base question answering (kbqa) system converts a natural language question to a logical query, which is executed on the kb to return the answer. the prevalence of diverse kbs, in open domain as well as in enterprises, makes kbqa a task of great practical significance. unfortunately, traditional supervised models @xcite @xcite @xcite @xcite @xcite and even recent llm-based fewshot in-context learning (fs-icl) architectures @xcite require large volumes of labeled (question, logical form) pairs for model training or for exemplar selection. such data is costly and time-consuming to annotate in real applications, and this remains the biggest hurdle for rapid and low-cost deployment of kbqa systems. a more realistic scenario for a new kb is that a domain expert provides a small number of questions and their logical forms as labels. in this paper, we ask whether this small data, in conjunction with a readily available and larger open-domain benchmark kbqa dataset @xcite @xcite seen as a source domain, can be used to bootstrap a kbqa system for the new kb, which forms the target domain. we propose this as the few-shot transfer learning setting for kbqa. to the best of our knowledge, this practically useful setting has not been studied in the kbqa literature. it differs from unsupervised transfer learning @xcite , which assumes a large volume of unlabeled training questions in the target domain. we first note that source and target tasks can differ along multiple dimensions. first, the two kbs may not have much overlap in schema or data. additionally, the distributions of logical forms and natural language questions may also be quite different. this makes it extremely challenging to transfer source-trained models to a target kbqa system, with limited target supervision. to handle this challenge, we resort to modern day llms, which have achieved unprecedented success in language understanding and code generation tasks, with only few-shot in-context learning (fs-icl). we present fusic-kbqa (fused supervised and in-context learning for kbqa), which uses source-trained models to construct llm prompts for fs-icl. in particular, given a target question, multiple source-trained retrievers with complementary strengths retrieve potentially relevant candidate paths, relations and entity types from the kb. the llm then performs re-ranking of this retrieved information to increase their target relevance. the re-ranked retrieval is provided in the fs-icl prompt of another llm call to generate logical forms. to minimize syntax errors, fusic-kbqa generates logical forms in a common lan-guage (sparql) instead of niche languages used in kbqa (e.g. s-expressions). to minimize semantic errors, fusic-kbqa provides executionguided feedback to the llm based on whether the logical form on execution returns an empty answer. we perform extensive experiments with multiple pairs of source-target datasets of varying complexity. we show that fusic-kbqa outperforms adaptations of sota supervised models and recent few-shot llm fs-icl models for kbqa by large margins. additionally, we find that even for indomain kbqa (non-transfer) with limited training data, fusic-kbqa outperforms sota supervised kbqa models, suggesting that it is robust across various low-data settings. our specific contributions are as follows. (a) we propose the important task of few-shot transfer learning for kbqa. (b) we develop a novel kbqa architecture that combines the benefits of supervised retrievers and llm prompting for generation and refinement. (c) we carefully create multiple kbqa transfer datasets of varying complexity, which we publicly release to aid future research 1 . (d) we demonstrate using extensive experiments that our fusion architecture has strong performance for few-shot transfer. (e) we additionally demonstrate benefits of this architecture for the in-domain kbqa setting with limited training data. ## related work in-domain supervised kbqa @xcite @xcite @xcite involves only a single domain with a large volume of in-domain labeled examples, which are used to train or fine-tune llm-based architectures. almost all sota approaches for this problem use a retrieve-then-generate architecture, where the first stage retrieves kb elements relevant to a question, and then provides these as context to a generation stage that uses an llm such as t5 to generate the logical form. pangu @xcite is different in that it fine-tunes llms to discriminate between relevant logical forms, and interleaves retrieval and discrimination step by step. retinaqa @xcite is a contemporaneous kbqa model for answerable as well as unanswerable questions @xcite ) that combines traversal-based kb retrieval with sketchfilling based construction. it also achieves sota 1 https://github.com/dair-iitd/fusic-kbqa/ performance for in-domain supervised kbqa. more recent in-domain kbqa approaches use few-shot in-context learning (fs-icl) with llms. these are still dependent on large volumes of indomain training data for selecting the most relevant few-shots. different from retrieve-then-generate, kb-binder @xcite and kb-coder @xcite follow a generate-then-ground architecture that first generates kb-independent partial programs using fs-icl, and then grounds these for the specific kb using unsupervised techniques. crucially, these do not take any advantage of the labeled in-domain examples using supervised learning. in recent work, shu and yu (2024) use tiara's supervised retriever followed by fs-icl llm generation. we improve over this basic architecture in multiple ways -by accommodating multiple retrievers, re-ranking retrieval results and including execution-based feedback. unsupervised transfer learning for kbqa @xcite involves large volumes of labeled training data in the source domain, and additionally a large volume of unlabeled data in the target domain. sota models for this setting follow the generate-then-ground architecture, again by first generating kb-independent (partial) programs, which are then adapted for the target kb using unlabeled examples. these models do not look to benefit from fs-icl using llms. llm prompting techniques related to ours have been used outside of kbqa. re-ranking of output from supervised or unsupervised retrieval models using llms has recently been used for information extraction @xcite and multi-hop question answering over documents @xcite . we are not aware of the use of llms for retrieval re-ranking for kbqa or for combining rankings from multiple retrievers. reflexion @xcite is a recent rl framework that transforms environment signals to language-based feedback for an llm. it explores python function generation as one of the use-cases, where output accuracy on generated test-cases is fed back to the llm for improvement. our execution-guided feedback can be imagined as a specific instance of this framework, but we are not aware of any use of similar ideas for generating logical forms for kbs. few-shot transfer learning for kbqa: a knowledge base g consists of a schema with data stored under it. the schema consists of entity types t and binary relations r defined over pairs of types. the data consists of entities e as instances of types t , and triples or facts f ⊆ e × r × e. we are given a target knowledge base g t (consisting of entity types t t , relations r t , entities e t and facts f t ) and a natural language question q t , and the goal is to generate a structured query or a logical form l t , which when executed over g t returns a non-empty answer a t for the question q t . in few-shot transfer learning, we are provided with target few-shots d t containing tens of labeled training examples of questions and logical forms in the target domain. in addition, a related source domain has a source knowledge base g s consisting of its own types t s , relations r s , entities e s and facts f s , and a much larger source training set d s containing thousands of labeled training examples of questions and corresponding logical forms. a special case of this setting is zero-shot transfer learning, where d @xmath0. proposed approach: retrieve-then-generate architectures for kbqa -used by state-of-the-art models for the supervised in-domain setting -first retrieve relevant kb-elements for the query and then use these as context for generating the logical form. the kb-retrievers used in such models have steadily grown in sophistication in recent years. on the other hand, generate-then-ground models for kbqa -used for unsupervised transfer learning and in domain few-shot learning with llms -first generate kb-independent incomplete or draft logical forms based on the question, and then ground these for the specific kb. for our solution, we assume that we have access to a black-box large language model. we use this term to refer to 100b+ parameter llms such as gpt-3 and beyond, which cannot be trained or fine-tuned for all practical purposes, and only offer access through prompting and in-context learning. in the rest of this paper, when we refer to llms, unless otherwise specified, we mean such models. we begin by asking: which of the two kbqa architectures above is better suited to simultaneously take advantage of a large source training set and an llm to accurately generate logical forms with limited target supervision? llms excel at understanding and generating language, but lack knowledge of the target kb. therefore, we hypothesize that an llm can generate complete logical forms accurately using in-context exemplars from the target, when provided with high quality context retrieved from the target kb. high quality retrieval can be achieved using one or more sophisticated supervised retrievers -trained using the large source training set, and optionally adapted for the target using few-shot fine-tuning -and then re-ranking for the target using llm prompting. based on this hypothesis, we propose fusic-kbqa, a retrievethen-generate architecture for the few-shot transfer learning problem for kbqa. it is depicted at a high-level in fig. 1 . ## experiments we now present experimental evaluation of fusic-kbqa for few-shot transfer learning which is our main focus. we also evaluate its usefulness for the limited training data in-domain kbqa setting. ## discussion our exploration of llm for kbqa led to many failed experiments. we discuss these briefly here. mention detection and entity disambiguation: we experimented with fs-icl with gpt-4 for mention detection. however, this performed worse than the supervised version both in the transfer setting (by 1 pct point) and in the limited training data (5%) in-domain setting (by 10 pct points). for entity disambiguation, we explored gpt-4 re-ranking of the entities returned by the supervised model. however, this did not improve performance. self-consistency and majority voting: this is reported to improve performance for in-domain kbqa with chatgpt @xcite . however, with gpt-4, generating with higher temperature led to little variance in the generated logical forms in our experiments, naturally rendering majority voting meaningless. exemplar selection: kb-binder @xcite ) and gf-llm @xcite benefit from dynamic exemplar selection using bm25. however, we observed only minor improvements beyond random exemplar selection and static exemplars. ## conclusions we have motivated and addressed the few-shot transfer learning setting for kbqa. we propose fusic-kbqa as a retrieve-then-generate architecture that benefits from kb-retrieval using one or more source-trained supervised retrievers, which are adapted for the target using llm-based reranking. such retrieval provides precise and relevant context to an llm that generates logical forms with few-shot in-context learning, further refined by execution-guided feedback. we demonstrate the usefulness of fusic-kbqa using carefully designed experiments over multiple @xmath1. we show that fusic-kbqa significantly outperforms adaptations of sota kbqa models, both supervised and using llm few-shot in-context learning, for fewshot transfer. we also demonstrate that fusic-kbqa outperforms sota supervised kbqa models in their own backyard -the in-domain setting -when the volume of training data is limited. an interesting future direction is addressing unanswerability of questions in the few-shot transfer setting @xcite .