Do I have the Knowledge to Answer? Investigating Answerability of Knowledge Base Questions (2023)

Abstract: When answering natural language questions over knowledge bases, missing facts, incomplete schema and limited scope naturally lead to many questions being unanswerable. While answerability has been explored in other QA settings, it has not been studied for QA over knowledge bases (KBQA). We create GrailQAbility, a new benchmark KBQA dataset with unanswerability, by first identifying various forms of KB incompleteness that make questions unanswerable, and then systematically adapting GrailQA (a popular KBQA dataset with only answerable questions). Experimenting with three state-of-the-art KBQA models, we find that all three models suffer a drop in performance even after suitable adaptation for unanswerable questions. In addition, these often detect unanswerability for wrong reasons and find specific forms of unanswerability particularly difficult to handle. This underscores the need for further research in making KBQA systems robust to unanswerability.

## introduction the problem of natural language question answering over knowledge bases (kbqa) has received a lot of interest in recent years @xcite @xcite @xcite @xcite @xcite . an important aspect of this task for real-world deployment is detecting answerability of questions. this problem arises for kbs due to various reasons, including schema-level and data-level incompleteness of kbs @xcite , limited kb scope, questions with false premises, etc. in such cases, a robust and trustworthy model should detect and report that a question is unanswerable, instead of outputting some incorrect answer. answerability is well studied for qa over unstructured contexts @xcite @xcite @xcite . however, there is no existing work on answerability for kbqa. benchmark kbqa datasets @xcite @xcite contain only answerable questions. we first identify how different categories of kb incompleteness (schema and data incompleteness) affect answerability of questions. then, using grailqa @xcite , one of the largest kbqa benchmark dataset, we create a new benchmark for kbqa with unanswerable questions, which we call grailqability, by deleting various elements from the kb to simulate scope and fact coverage limitations. this involves addressing a host of challenges, arising due to different ways in which kb element deletion affects answerability of questions, dependence between deletion of different types of kb elements, the shared nature of kb elements across questions, and more. we also define and include different generalization scenarios for unanswerable questions in the test set, namely iid and zero-shot, mirroring those for answerable questions. we then use grailqability to evaluate the robustness of three recent state-of-the-art kbqa models, rng-kbqa @xcite , retrack @xcite and tiara @xcite , against unanswerable kb questions. we find that all three models suffer an overall drop in performance with unanswerable questions, even after appropriate adaptation for unanswerability via retraining and thresholding. more alarmingly, these often detect unanswerability for incorrect reasons, raising concerns about trustworthiness. additionally, while the strength of these models is that they learn at the schema-level, we find that this also results in significantly poorer ability to detect data-level incompleteness. using error analysis, we identify important failure points for these models. all of these highlight robustness issues for kbqa models in real applications, raising important questions for future research. in summary, our contributions are as follows. (a) we motivate and introduce the task of detecting answerabilty for kbqa. (b) we create grailqability, which is the first benchmark for kbqa with unanswerable questions. (c) using experiments and analysis on grailqability with three state-ofthe-art kbqa models , we identify aspects of unanswerability that these models struggle to identify. we release code and data for further research. 1 ## kbqa with answerability detection a knowledge base (kb) (also called knowledge graph) g contains a schema s (or ontology) with entity types (or types) t and relations r defined over pairs of types, which we together refer to as schema elements of the kb. the types in t are often organized as a hierarchy. it also contains entities e as instances of types, and facts (or triples) f ⊆ e × r × e, which we together refer to as data elements of the kb. the top layer of fig. 1(a ) shows example schema elements, while the bottom layer shows entities and facts. in knowledge base question answering (kbqa), we are given a question q written in natural language which needs 1 https://github.com/dair-iitd/grailqability. git to be translated to a logical form (or query) l that executes over g to yield a set of answers a. different logical forms, sparql (yih et al., 2016b), sexpressions @xcite , programs @xcite , etc., have been used in the kbqa literature. we concentrate on s-expressions @xcite , which employ set-based semantics and functions with arguments and return values as sets. these can be easily translated to kb query languages such as sparql, and provide a balance between readability and compactness @xcite . we call a logical form valid for g if it executes over g without an error. on successful execution, a logical form traces a path in the kb leading to each answer. fig. 1 (a) shows an example query with a valid logical form (using s-expression) and the path traced by its execution. we define a question q to be answerable for a kb g, if (a) q admits a valid logical form l for g, and (b) l returns a non-empty answer set a when executed over g. the example question in fig. 1(a ) is answerable for the shown kb. the standard kbqa task over a kb g is to output the answer a, and optionally the logical form l, given a question q, assuming q to be answerable for g. most recent kbqa models @xcite are trained with questions and gold logical forms. other models directly generate the answer @xcite . different train-test settings have been explored and are included in benchmark kbqa datasets @xcite . for a question q, let s q denote the schema elements in the logical form for q. given a training set q tr , a test question q is labelled iid if it follows the distribution for questions in q tr , and contains only schema elements seen in train s q ⊆ s qtr (we have overloaded notation to define s qtr ). alternatively, a test question q is labelled zero shot if it involves at least one unseen schema element (s q ̸ ⊆ s qtr ). finally, test question q involves compositional generalization if s q ⊆ s qtr but the specific logical form for q does not match that for any q ′ ∈ q tr . by negating the above answerability definition, we define a question q to be unanswerable for a kb g if (a) q does not admit a valid logical form l for g, or (b) the valid l when executed over the g returns an empty answer. clearly, meaningless and out-of-scope questions for a kb are unanswerable. even for a meaningful question, unanswerability arises due to incompleteness (in data or schema) in g. such questions admit an 'ideal kb' g * for which q has a valid ideal logical form l * which executes on g * to generate a non-empty ideal answer a * . the available kb g lacks one or more schema or data elements making q unanswerable. fig. 1(a ) illustrates an available kb, with missing elements with respect to the ideal kb shown in red. in fig. 1(b ), questions 1-2 yield valid queries for the available kb but missing facts lead to empty answers, while questions 3-5 lack schema elements for valid queries. the task of kbqa with answerability detection, given a question q and an available kb g, is to (a) appropriately label the answer a as na (no answer) or the logical form l as nk (no knowledge, i.e., query not possible) when q is unanswerable for g, or (b) generate the correct non-empty answer a and valid logical form l when q is answerable for g. the training set may now additionally include unanswerable questions labeled appropriately with @xmath0. note that training instances do not contain 'ideal' logical forms for the unanswerable questions that have @xmath1. mirroring answerable questions, we define different train-test scenarios for unanswerable questions as well. an iid unanswerable question in test follows the same distribution as unanswerable questions in train, and all missing kb elements (schema elements in its ideal logical form and missing data elements in its ideal paths) are encountered in train unanswerable questions associated with the same category of incompleteness. for example, the missing schema element research area for the first test question in fig. 1(c ) is covered by the second train question. in contrast, a zero-shot unanswerable test question involves at least one missing kb element (schema element in its ideal logical form or data element in its paths) that is not part of any unanswerable question in train associated with same category of incompleteness. e.g., the missing schema elements (located in and works at) for the second and third test questions in fig. 1(c ) are not covered by any unanswerable question in train. we further define two sub-classes, partial and complete zeroshot, for zero-shot unanswerable questions, but for clarity, discuss these in sec. 5. ## grailqability: extending grailqa with answerability detection in this section, we describe the creation of a new benchmark dataset for kbqa with unanswerable questions. in a nutshell, we start with a standard kbqa dataset containing only answerable questions for a given kb. we introduce unanswerability in steps, by deleting schema elements (entity types and relations) and data elements (entities and facts) from the given kb. we mark questions that become unanswerable as a result of each deletion with appropriate unanswerability labels. we control the percentage of questions that become unanswerable as a result of each type of deletion. many complications arise in this. (a) deletion of different kb elements affect answerability differently. some affect logical forms and answers, while others affect answers only. (b) the same kb element potentially appears in paths or logical forms of multiple questions. (c) kb elements cannot be deleted independently -entity types are associated with relations and entities, while relations and entities are associated with facts. (d) questions with multiple answers remain answerable until the fact paths to all of these answers have been broken by deletions. (e) choosing kb elements to delete uniformly at random does not resemble incompleteness in the real world. we address these issues as follows. (a-b) we iterate over the 4 categories of kb elements to be dataset #q #lf #d #r #t #e q. type test scenarios a u a u grailqa 64,331 4969 86 3720 1534 32,585 ✓ ✗ i, c, z ✗ grailqability 50,507 4165 81 2289 1081 22,193 ✓ ✓ i, c, z i, z grailqability dataset: we make use of grailqa @xcite , which is one of the largest and most diverse kbqa benchmark based on freebase but contains only answerable questions, and create a new benchmark for kbqa with answerability detection. we call this grailqability (grailqa with answerability). we make this dataset public. aligning with earlier qa datasets with unanswerability @xcite @xcite , we keep the total percentage of unanswerable questions as 33%, splitting this nearly equally (8.25%) between deleted entity types, relations, entities and facts. train-test split: since the test questions for grailqa are unavailable, we use the train and dev questions. we keep aside the compositional and zero shot questions from dev as the compositional and zero shot answerable questions in our new dev and test set. we then combine the train and iid dev questions, introduce unanswerability into these by running the 4 categories of deletion algorithms in sequence, and split these to form the new train and iid test+dev (both answerable and unanswerable) and zero shot unanswerable test+dev questions. the unanswerable questions in test and dev contain 47% iid, and 53% zero-shot. statistics for grailqability and grailqa are compared in tab. 1. sizes of the different splits are shown in tab. 2. details on dataset creation are in appendix (a.1). ## experimental setup kbqa models: among state-of-the-art kbqa models, we pick rng-kbqa @xcite , retrack @xcite and tiara @xcite . these report state-of-the-art results on grailqa as well as on webqsp @xcite @xcite )the two main benchmarks. on the grailqa leader board, 2 these are the top three published models with available code (at the time of submission). since these generate logical forms, we expect these to be more robust to data level incompleteness than purely retrieval-based approaches @xcite @xcite @xcite . rng-kbqa (ye et al., 2022) first uses a bertbased @xcite ranker to select a set of candidate logical forms for a question by searching the kb, and then a t5-based @xcite model generates the logical form using the question and candidates. retrack @xcite also uses a rank and generate approach, but uses a dense retriever to retrieve schema elements for a question, and grammar-guided decoding using an lstm @xcite to generate the logical form using the question and retrieved schema items. tiara @xcite combines the retrieval mechanisms of the first two models to include both candidate logical forms as well as candidate schema elements from the kb. it then uses constrained decoding like retrack but using t5 @xcite . all three models use entity disambiguation to find kb entities mentioned in a question and also check execution for generated logical forms. ## results and discussion we structure our discussion of experimental results around four research questions. ## conclusions and discussion we have introduced the task of detecting answerability when answering questions over a kb. we have released grailqability 1 as the first benchmark dataset for kbqa with unanswerable questions, along with extensive experiments on three kbqa models. we find that no model is able to replicate its answerable performance for the unanswerable setting even with appropriate retraining and thresholding, though both these methods of adaptation help in improving performance substantially. further, we find that there is a trade-off between robustness to missing schema elements and missing data elements. the models find schema-level incompleteness easier to handle while data-level incompleteness substantially affects the models that enforce data-level constraints while decoding. another observation is that the models get quite confused for those unanswerable questions that contain a schema element seen in an answerable train question, along with a missing schema element that is not seen at training. finally, while tiara turns out to be the best overall model, different models find different categories of unanswerability to be more challenging. this suggests that new kbqa models will need to combine architectural aspects of different existing models to best handle unanswerability. we believe that our dataset and observations will inspire research towards developing more robust and trustworthy kbqa models. by iit delhi. he also acknowledges travel support from google and yardi school of ai travel grants. we thank the iit delhi hpc facility for its computational resources.