Iterative Repair with Weak Verifiers for Few-shot Transfer in KBQA with Unanswerability (2025)

Abstract: Real-world applications of KBQA require models to detect different types of unanswerable questions with a limited volume of in-domain labeled training data. We propose the novel task of few-shot transfer for KBQA with unanswerable questions. The state-of-the-art KBQA few-shot transfer model (FuSIC-KBQA) uses an iterative repair strategy that assumes that all questions are answerable. As a remedy, we present FUn-FuSIC – a novel solution for our task that extends FuSIC-KBQA with Feedback for Unanswerability (FUn), which is an iterative repair strategy for answerable as well as unanswerable questions. FUn uses feedback from a suite of strong and weak verifiers, and an adaptation of self-consistency for unanswerability for assessing answerability of questions. Our experiments show that FUn-FuSIC significantly outperforms suitable adaptations of multiple LLM-based and supervised SoTA models on our task, while establishing a new SoTA performance for answerable few-shot transfer as well. We have made datasets and other resources publicly available at https://github.com/dair-iitd/funfusic/.

## introduction the semantic parsing formulation of the knowledge base question answering (kbqa) task takes as input a knowledge base (kb) and a natural language question, and outputs a logical form (or program) that produces the answer upon execution over the kb. kbqa has important real-world applications, which require kbqa systems to be low-resource (i.e., trained only with a few taskspecific labeled examples), and robust, specifically able to identify questions that cannot be answered from the kb. traditional supervised models (e.g., @xcite @xcite ) and even recent llm few-shot in-context learning (fs-icl) architectures @xcite for kbqa fall short in both aspects. limited recent work has addressed these independently -indomain methods for kbqa with unanswerability trained with large labeled data @xcite , and fusic-kbqa for few-shot transfer assuming answerable questions @xcite . no existing single kbqa model simultaneously addresses both desiderata. in response, we propose the novel task of fewshot transfer learning for kbqa with unanswerability. specifically, the target domain has only a few labeled examples of answerable and unanswerable questions, while the source domain has thousands of labeled examples, but containing only answerable questions. for few-shot kbqa transfer, fusic-kbqa uses a retrieve-then-generate framework: retrieval of relevant schema and kb snippets followed by an llm-based generation and a subsequent iterative execution-error-guided repair. specifically, multiple feedback-guided repair iterations are executed, checking emptiness of answers obtained by executing the generated program as indication of correctness, until a non-empty answer is obtained. this naturally fails when questions are allowed to be unanswerable. a simple-fix for addressing unanswerability (fusic-kbqa-u) is to drop the inappropriate repair step, and modify the llm prompt to accommodate unanswerable questions, along with relevant in-context exemplars. unlike in studies for unanswerability in general qa @xcite , we found that fusic-kbqa-u mostly generates incorrect logical forms for unanswerable questions. as a remedy, we design a novel solution: fun-fusic (feedback for unanswerability in fusic-kbqa). the key idea is to modify iterative repair, which earlier relied on a single strong verifier for the logical form's incorrectness, to rely on a suite of strong and weak verifiers, where strong veri-fiers identify certain errors, whereas weak verifiers identify potential errors in the current logical form. fun-fusic's verifiers consider both the logical form and the answer. for answers, non-emptiness check is now a weak verifier, given potentially unanswerable questions. for logical forms, we use strong verifiers to identify obvious syntactic and semantic errors. we also propose a novel verifier involving a 3-component llm-based pipeline: nonequivalence of the original question and the backtranslation of the logical form. this verifier is also weak, due to potential errors in back-translation as well as in equivalence classification. using such iterative strong and weak verification based repair, fun-fusic constructs a set of candidate logical forms. for selecting the consensus logical form from this set, using the majority answer as in selfconsistency @xcite breaks down in the face of unanswerability. we introduce selfconsistency for unanswerability, which assesses the likelihood of the majority answer, empty or otherwise, to select the consensus logical form. since no datasets exist for our novel task, we create two new datasets for kbqa transfer with unanswerable questions. our experiments show that fun-fusic comprehensively outperforms different categories of sota models suitably adapted for this task, including llm-based and more traditional models. we further find that iterative repair of logical forms using weak verifiers holds promise for even for kbqa with only answerable questions. using experiments over benchmark datasets for this task, we show that the restriction of fun-fusic for the answerable setting improves upon the sota model for the task. in summary, our specific contributions are as follows. (a) we propose the problem of few-shot transfer for kbqa with unanswerability. (b) we present fun-fusic that uses iterative repair with error feedback from a diverse suite of strong and weak verifiers. (c) we create new datasets for the proposed task, which we make public. (d) we show that fun-fusic outperforms adaptations of sota kbqa models for this new task. (e) we also show that even for answerable-only kbqa, fun-fusic outperforms the corresponding sota model. ## related work in-domain kbqa using supervised models @xcite @xcite @xcite @xcite @xcite and using llm few-shot approaches @xcite @xcite is well explored in literature. these use high volumes of labeled data, either for training or selecting the most relevant few shot exemplars. for in-domain kbqa, unanswerability has recently been studied @xcite . @xcite create the grailqability dataset with different categories of unanswerability, and show the inadequacy of superficial adaptations of answerable-only kbqa models. retinaqa @xcite is the sota model for kbqa unanswerability. however, this also requires large volumes of training data. for kbqa transfer @xcite , low-resource was originally not a focus. more recently, few-shot transfer for kbqa has been addressed by fusic-kbqa @xcite . fusic-kbqa uses a retrievethen-generate framework with an llm-based generation stage with iterative error feedback based repair. however, this formulation assumes answerability of all questions. simple llm prompting techniques have been used to address unanswerability outside of kbqa @xcite , but without any notion of feedback or iterative repair. other approaches @xcite use execution based refinement for program generation but without any notion of unanswerability. fun's iterative repair idea may be useful in other natural language to program generation tasks where non-existence of a program with the required specification has not been studied to the best of our knowledge, such as nl-to-sql @xcite and program self-repair using llms @xcite @xcite . ## background & problem definition a knowledge base (kb) g consists of a schema and data. the schema consists of entity types (or classes) t and binary relations r defined over pairs of types. the data consists of entities e as instances of types t , and triples or facts f ⊆ e × r × e. given a target kb g t and a natural language question q t , the basic kbqa task is to generate a structured query or logical form l t (in a kb query language, such as sparql), which when executed over g t returns an answer a t (a t ⊂ e in general). other than sparql @xcite , niche languages such as s-expressions @xcite are commonly used for logical forms in kbqa. in supervised in-domain kbqa, the target has large volumes of labeled training examples of questions and associated logical forms. for few-shot in-domain kbqa in contrast, target few-shots d t contain tens of labeled training examples. in few-shot transfer learning for kbqa @xcite , a related source domain has a source kb g s (with its own types, relations, entities and facts), and a larger source training set d s with thousands of labeled training examples. following @xcite , a question q is answerable for a kb g if it admits a corresponding logical form l which when executed over g returns the ideal non-empty answer a. a question is unanswerable if it either (a) does not have a valid logical form for g (schema-level unanswerability), or (b) it has a valid logical form l for g, but l returns an empty answer upon execution on g, different from the ideal non-empty answer (data-level unanswerability). schema-level unanswerability arises due to missing types and relations, while missing entities and facts lead to data-level unanswerability. however, absence in g of any entity mentioned in the question is categorized as schema level unanswerability, since it invalidates the logical form. more details are in appendix a.1.2. in kbqa with unanswerability @xcite , given a question q, the model needs to output (a) a logical form l and a non-empty answer a for answerable q, (b) @xmath0, or (c) a valid logical form l and @xmath1. in the supervised in-domain setting, this task involves large volumes of labeled training questions, containing both answerable and unanswerable, for the target. we now define our problem of interest: few-shot transfer learning for kbqa with unanswerability. a target question q t may be answerable or unanswerable due to missing schema or data in the target kb g t . target few-shot examples d t contain both answerable and unanswerable questions of different categories. the source training data d s has large volumes of labeled training data. considering real world constraints, where most kbqa datasets contain only answerable questions, we assume that d s contains only answerable questions. compared to the earlier few-shot kbqa transfer task definition, now there is additionally an unanswerability mismatch between the source and the target distribution. more details are in the appendix (sec. a.1). ## proposed approach: fun-fusic our proposed model fun-fusic preserves the basic architecture of fusic-kbqa and adapts its iterative repair strategy for unanswerability. the high-level algorithm is described in algo. 1. (since the algorithms are not specific to the transfer task, we use q instead of q t for brevity.) preserving the retrieve-then-generate framework of fusic-kbqa, the retrieval stage (line 2) performs kb retrieval for q t using a set r of one or more supervised retrievers. each retriever r i is sourcetrained and further target fine-tuned if required. the retrieval output r of each r i consists of relevant schema elements (types and relations) for q t , and data paths emanating from mentioned entities in q t . the union of these, along with q t , is fed to the generation stage, which uses prompting with an llm l to generate logical forms using the target few-shots d t . more details of the retrieval stage are in the appendix (sec. a.8.1). fun-fusic differs from fusic-kbqa in its iterative repair strategy in lines 3, 4 and 6 of algo. 1. the llm generation instruction is modified to admit the possibility of unanswerability, and the few shots are modified to include examples of unanswerable questions. however, this simple approach is error-prone. so, we bias the instruction towards one type of error. specifically, when uncertain we now come to the more significant modifications. first, l (0) is iteratively repaired using feedback as before, but this step is adapted for unanswerability. this iterative repair, which we name feedback for unanswerability (fun) (line 4), either confidently outputs a single logical form l (with corresponding answer a) (line 5) or generates a set l of candidate logical forms, which is further analyzed for a consensus logical form and answer. for this, we introduce self-consistency for unanswerability (scun) (line 6). scun assesses the likelihood of the majority answer in l, empty or otherwise, to produce the final output. in the rest of this section, we describe fun and then scun. fig. 1 illustrates flow of fun and scun using examples. a real example of fun execution is in sec. a.12. the fun algorithm is described in algo. 2 start-ing with the initial logical form l (0) , fun performs at most n verify-and-repair iterations to create a candidate set l of probable logical forms. fig. 1 shows 3 fun iterations for kb1 and kb2, and 2 for kb3. in the i th iteration, fun generates a new logical form l (i) by prompting l using q t and feedback f received from checks in all previous iterations (line 13). l (i) goes through a sequence of verifications. fun uses two sets of verifiers. the strong verifiers v s are guaranteed to be correct, while the weak verifiers v w are potentially erroneous. k 1 and k 2 denote the total number of strong and weak verifiers respectively. the specific verifiers that we use in this paper are defined later in the section. a template-based feedback string f is appended to the generation prompt for l (i+1) based on the specific verifier that l (i) failed. if l (i) fails a strong verifier, it is rejected (line 6). in the example, this happens for all three kbs in iteration 1. if l (i) passes all checks, strong and weak (line 15), fun terminates by outputting (@xmath2, @xmath3, where a (i) is the answer obtained by executing this happens in iteration 3 for kb3. otherwise, if l (i) passes at least one weak verifier but not all, it is added to candidate logical form set l (line 11). this happens for iterations 2, 3 and 4 for kb1 and kb2. logical form verifiers: fun uses a suite of verifiers, categorized as strong (v s ) and weak (v w ). these may be syntactic, semantic, or executionbased, defined using simple rules or complex llm functions over l, q and g. note that unlike unit tests in program synthesis, the verifiers do not have knowledge of the gold logical form, the gold answer or answerability of the question. we now briefly describe the specific verifiers that we use for this paper. additional details about the verifiers are in the appendix (sec. a.11.2 and sec. a.3). note that fun is a framework that is capable of working with a wholly different suite of meaningful verifiers. identifying candidate logical forms: unless some logical form passes all checks and is therefore returned (algo. 2 line 15), fun constructs a candidate set l of logical forms that are potentially flawed but not certainly so. for our specific suite of weak verifiers, l (i) is added to l if it passes one of v4b (a is non-empty) as for lf3 for kb1, or v3 (l i is equivalent to q), as for lf3 for kb2. ## experiments we now present experimental evaluation of fun-fusic. first, for few-shot kbqa transfer with unanswerability, we address the following research questions. (r1) how does fun-fusic compare against sota kbqa models suitably adapted for this setting? (r2) how does fun-fusic perform across different categories of unanswerability? (r3) how do the different components of fun-fusic contribute to its performance? then, for answerable kbqa few-shot transfer, we ask: (r4) how does fun-fusic compare against sota kbqa models for this setting? ## conclusions for real-world robust and low-resource kbqa, we have proposed the novel task of few-shot transfer learning with unanswerability. we have introduced a new notion (fun) of iterative feedback guided repair for answerable as well as unanswerable questions. fun (i) uses feedback from a diverse suite a strong and weak verifiers -including a novel back-translation based verifier -to create a set of candidate logical forms, and (ii) assesses this candidate set to either to detect unanswerability (and its category) or identify the best logical form using self consistency adapted for unanswerability (scun). we propose fun-fusic that replaces the existing the iterative strategy, that assumes answerability of questions, with fun in the sota few-shot answerable-only kbqa transfer model (fusic-kbqa). using two newly created datasets for this novel task, we show that fun-fusic significantly outperforms adaptations of fusic-kbqa and other sota models for this setting, and also for answerable few-shot transfer kbqa. our error analysis suggests that performing well across categories of unanswerability for few-shot transfer is still a challenge for kbqa and should be a focus of further research. we have made our datasets and other resources public foot_1 .