R etina QA : A Robust Knowledge Base Question Answering Model for both Answerable and Unanswerable Questions (2024)

Abstract: An essential requirement for a real-world Knowledge Base Question Answering (KBQA) system is the ability to detect the answerability of questions when generating logical forms. However, state-of-the-art KBQA models assume all questions to be answerable. Recent research has found that such models, when superficially adapted to detect answerability, struggle to satisfactorily identify the different categories of unanswerable questions, and simultaneously preserve good performance for answerable questions. Towards addressing this issue, we propose RetinaQA, a new KBQA model that unifies two key ideas in a single KBQA architecture: (a) discrimination over candidate logical forms, rather than generating these, for handling schema-related unanswerability, and (b) sketch-filling-based construction of candidate logical forms for handling data-related unaswerability. Our results show that RetinaQA significantly outperforms adaptations of state-of-the-art KBQA models in handling both answerable and unanswerable questions and demonstrates robustness across all categories of unanswerability. Notably, RetinaQA also sets a new state-of-the-art for answerable KBQA, surpassing existing models. We release our code base for further research: https://github.com/dair-iitd/RetinaQA.

## introduction question answering over knowledge bases (kbqa) @xcite @xcite @xcite @xcite @xcite @xcite requires answering natural language questions over a knowledge base (kb), most commonly via generating formal queries or logical forms that are then executed over the knowledge base to retrieve the answers. when users interact with kbs in real-world settings, unanswerability of questions arises naturally. users are typically unfamiliar with the schema and data of the underlying kb. further, kbs are also often incomplete. while specialized models for handling unanswerability have been proposed for other question answering tasks @xcite @xcite @xcite , all existing models for kbqa assume answerability of questions over the given kb. recently, @xcite published a benchmark dataset called grailqability incorporating different categories of unanswerable questions. this work also demonstrated that state-of-the-art kbqa models perform poorly off-the-shelf for unanswerable questions. this performance improves with superficial adaptations for unaswerability, such as adding unanswerable questions during training and thresholding. however, such adaptations significantly hurt performance for answerable questions. additionally, different state-ofthe-art models struggle with different categories of unaswerability, such as (a) questions for which schema elements (i.e. relations or entity types) are missing in the kb, and which therefore do not have valid logical forms, and (b) questions for which data elements (i.e. entities or facts) are missing in the kb, and which therefore have logical forms that are valid, but return empty answers on execution. this work demonstrated that there is no single model which performs well for all categories of unanswerable questions and answerable ones. our analysis of existing models reveals two key drawbacks that make kbqa models less robust -imperfect calibration and over-reliance on pathbased retrieval. to overcome these, we propose a new multi-stage retrieve, generate, and rank architecture for kbqa named retinaqa, which brings together several ideas from the literature. first, a robust kbqa model has to separate questions that are answerable (having a valid logical form) from those that are unanswerable due to missing schema elements (not having a valid logical form). this requires good model calibration. most kbqa models use generation of logical form as their final step, which has calibration challanges. instead, since discriminative models are generally better calibrated, retinaqa's final stage discriminates among candidate logical forms, better separating questions without any logical form. secondly, existing models assume answerability, where all logical forms have accompanying paths in the kb. these models learn to rely on retrieved paths from the kb to generate logical forms. however, missing data elements may break any valid path, while the question, even though unanswerable, still has a valid logical form. existing models falter here, and end up generating some other logical form corresponding to paths that do exist in the kb. to handle such unanswerable questions, retinaqa additionally employs sketchfilling-based construction. it first generates schemaindependent high-level sketches, and then grounds these for the specific kb using relevant schema elements retrieved for the question. using experiments over grailqability, we demonstrate that the performance of retinaqa is not only stable but also significantly better than that of adaptations of multiple state-of-the-art kbqa models that assume answerability, not only across different categories of unanswerable questions, but also for answerable ones. interestingly, we demonstrate that the retinaqa architecture performs strongly for fully answerable kbqa benchmarks as well, and establishes a new state-of-the-art performance on the grailqa dataset. ## related work the predominant approach for supervised kbqa uses the question to construct a logical form, which is then executed to retrieve the answer @xcite @xcite . state-of-the-art models use k-hop path traversal to retrieve data paths from linked entities in the question @xcite . some models, instead @xcite or additionally @xcite , retrieve schema elements (namely, entity types and relations) based on the question. these utilize transformer-based decoders to generate the target logical forms. we have observed that generative models are not adequately calibrated to separate answerable and unanswerable questions correctly. in contrast, pangu @xcite uses language models to incrementally enumerate and discriminate between partial logical forms. though this approach is better calibrated, its path-level enumeration makes it brittle for questions with missing data elements. some retrieval-based methods @xcite also perform ranking of answer paths, but directly select answer nodes. these methods maximize the similarity score between a relation and the question. adapting such techniques to detect unanswerable questions is difficult. in contrast, we perform contrastive-learning-based one-shot discrimination on fully-formed logical form candidates in the final stage. in addition to supervised in-domain settings, transfer @xcite and few-shot @xcite settings have also been studied for kbqa. here, test questions involve unseen kb relations and entity types. approaches for these settings first generate high-level sketches (also called drafts or skeletons) that capture the syntax of the target language. these sketches are then filled in with kb-specific arguments to construct complete programs, which are finally scored and ranked. notably, these transfer architectures do not involve any traversal-based component to retrieve logical forms. in contemporaneous work on few-shot transfer, @xcite propose the fusic-kbqa framework accommodating one or more supervised retrievers coupled with an llm for retrieval reranking and logical form generation. this demonstrates how retrievers from existing supervised models @xcite can be adapted and augmented using llms for low-supervision settings. in principle, retinaqa can also be accommodated as a retriever in this framework. unanswerability and specialized models for detecting unanswerable questions have been studied for many question-answering tasks @xcite @xcite @xcite . however, no specialized models have been proposed for detecting unanswerable questions in kbqa. all existing kbqa models assume that questions have valid logical forms with non-empty answers. even in the transfer setting for kbqa @xcite , though the target logical forms may involve schema elements unseen during training, questions are still assumed to be answerable. recent work @xcite figure 1 : retinaqa architecture showing different components illustrated with an example question. symbols l and s represent candidate logical form and its score as discriminator output, l the output logical form, a the final answer, l(1) the top ranked logical form, ex(l) the answer obtained by executing logical form l. nk and na are special symbols indicating no knowledge (for logical form) and no answer. the logical form in red under lf retriever would not be found if data element (c. manning, works at, stanford) is missing in the kb, and additionally that in red under lf integrator would not be found if relation works at is missing in kb schema and therefore not retrieved by the schema retriever. the candidate logical form in red under discriminator would not be found if both of these are missing. ## the problem and our solution we first briefly define the kbqa with unanswerability task, and then describe the architecture of our proposed retinaqa model. ## experiments we address the following research questions: (1) how does retinaqa compare against baselines for answerable and unanswerable questions in two different training settings: one with only answerable questions and another with both answerable and unanswerable questions? (2) how does retinaqa perform for different categories of unanswerable questions? (3) how does retinaqa perform in the answerable-only setting? (4) to what extent do the different components of retinaqa contribute towards its performance in (1), ( 2 ) and (3) above? ## conclusions we have presented retinaqa, the first specialized kbqa model that shows robust performance for both answerable and unanswerable questions. to address this challenging task, retinaqa identifies candidate logical forms using data-traversal-based retrieval, as well as schema-based generation via sketch-filling that bridges over data gaps that break traversal. retinaqa also discriminate between fully formed candidate logical forms at the final stage instead of generating these. this enables it to better differentiate between valid and invalid logical forms. in doing so, retinaqa unifies key aspects of different existing kbqa models that assume answerability in iid and transfer settings. unlike superficial adaptations of existing sota models for unanswerability, we show that reti-naqa demonstrates stable performance across adaptation strategies, question categories (answerable and different categories of unanswerable questions) and different generalization settings. we also show that retinaqa performs significantly better for unanswerable questions and almost at par for answerable ones. retinaqa also performs well with only answerable training, which is a likely real-world scenario. retinaqa also retains this stability and performance for answerable-only kbqa benchmarks, achieving a new state-of-the-art performance on the answerable-only grailqa dataset. we release our code-base foot_2 for further research.