Exploring the Reasons for Non-generalizability of KBQA systems (2023)

Abstract: Recent research has demonstrated impressive generalization capabilities of several Knowledge Base Question Answering (KBQA) models on the GrailQA dataset. We inspect whether these models can generalize to other datasets in a zero-shot setting. We notice a significant drop in performance and investigate the causes for the same. We observe that the models are dependent not only on the structural complexity of the questions, but also on the linguistic styles of framing a question. Specifically, the linguistic dimensions corresponding to explicitness, readability, coherence, and grammaticality have a significant impact on the performance of state-of-the-art KBQA models. Overall our results showcase the brittleness of such models and the need for creating generalizable systems.

## introduction the task of question answering over knowledge bases (kbqa) involves answering a natural language question by querying a predefined knowledge base (kb). while progress in kbqa research has addressed several challenges like answering complex questions, multi-hop reasoning @xcite , conversational qa @xcite , and multi-lingual kbqa @xcite , most of the prior work in this field has been restricted to an i.i.d. setting @xcite . in a real-world setting, a kbqa system should be well-equipped to handle users' queries that were unseen during training. to motivate research along this front, @xcite proposed a dataset (grailqa) with an associated leaderboard to benchmark the generalizability of kbqa methods to new compositions, and unseen schema items (zeroshot). multiple state-of-the-art models @xcite @xcite have achieved remarkable performance on the zero-shot split giving the impression that kbqa generalization might be a solved problem. however, a cross-dataset evaluation of the models trained on grailqa reveals that they do not transfer well even for the more simpler one or twohop questions. we observe that while these models achieve impressive performance on the grailqa zero-shot (grailqa z) split, they fail to generalize to questions from other datasets like webqsp @xcite , graphq @xcite , and com-plexwebquestions @xcite ) even though they are built upon the same knowledge base (i.e. freebase). in this work we closely inspect the reasons for this drop. we analyse the structural and linguistic differences between questions from the different publicly available kbqa benchmark datasets. we observe that while structural complexity somewhat explains the performance variations across questions within the same dataset, it does not explain the performance drop when testing on other datasets. our analysis shows that the linguistic differences like explicitness and length of questions, grammaticality, readability, and coherence account for the degradation in performance. although webqsp and grailqa share the same underlying kb, the substantial differences in the annotation process manifests as samples having different linguistic properties. we find that these linguistic variations act as an additional dimension for evaluating the generalizability and real-world usefulness of kbqa systems. ## datasets in order to understand the zero-shot efficacy of the state-of-the-art kbqa models, we look at their performance on the following datasets: grailqa @xcite base relations. it's development and test sets have three splits to independently measure the i.i.d, compositional and zero-shot capabilities of kbqa systems. we leverage their publicly available training and dev sets for our experiments. webqsp @xcite contains questionanswer pairs from non-experts collected using the google suggest api, and uses amazon mechanical turk to get the answers for the obtained questions. graphq @xcite has varying question characteristics that include complexity along the semantic structure, qualitative analysis over answer space, topic of the question, and the number of possible answers for the questions. complexwebquestions (cwq) (talmor and berant, 2018a) builds on top of webqsp and automatically creates complex questions that include phenomena such as function composition, conjunctions, superlatives and comparatives. we consider these datasets for our experiments as all of them use freebase as their underlying kb. creating zero-shot splits: we categorize questions in the test/dev splits of the corresponding dataset into (i) non zero-shot (i.i.d. + compositional) and (ii) zero-shot similar to the categories proposed by @xcite . specifically, zeroshot instances have at least one schema item (class or relation) that were not seen during training in the original grailqa dataset. we note the criteria to be a bit lenient for relations whose corresponding inverse relation occurred during training (ex: inventors.inventions as opposed to inventions.invented_by). consequently, we update the zero-shot criterion to exclude questions where ei- reasoning paths: we characterize the complexity of the questions for different datasets based on the notion of reasoning paths as defined in @xcite . a reasoning path (hereforth rp) represents the sequence of actions (specifically relations traversed from the starting constraint(s) in the query graph) to reach the final answer. they provide a unified way to measure the complexity in terms of the number of hops and the number of constraints (examples shown in overall results: as shown in number of zero-shot schema items (#z): previous works @xcite have shown a degradation in performance of kbqa systems when exposed to unseen schema items. we thus compare the number of zero-shot schema items in the questions across the datasets. we observe that the zero-shot splits of the diferent datasets contain similar or fewer zero-shot schema items than grailqa z across the different reasoning paths ( controlling for rps, none of the other datasets have significantly more zero-shot items than grailqa z, suggesting that these questions are not necessarily more difficult, and the nongeneralizability of the evaluated systems cannot be solely attributed to this factor. ## analyzing linguistic variation in this section, we explore whether the regression in performance can be explained via the linguistic variation among the different kbqa datasets. we analyze the questions in these datasets using the dimensions discussed below: sentence length (#w): firstly, we compare the length of the natural language questions in each dataset. we find that webqsp seems to have the shortest questions ( ## discussion overall, our results show that systems trained on grailqa seem to transfer the best to graphq which has similar linguistic properties to grailqa i.e., higher sentence lengths and number of common nouns, medium formality scores, and lower readablity. this is inline with the similarity in their annotation processes that requires annotators to refer to a query graph to arrive at a nl question, which might bias them to include hidden nodes in the reasoning path. the questions in grailqa are more explicit (highest #n) than graphq. compare this with the extremely poor performance on webqsp, which can be explained by the stark differences in the language used in this dataset i.e., lesser (i) number of words in question sentences, (ii) number of common nouns and (iii) formality, and higher readability. this follows from webqsp containing real-world non-expert queries collected from a search engine. finally, despite cwq having longer questions like grailqa, it does not contain as many #n suggesting that the annotators do not rely on introducing hidden classes in the nl question while merging the simpler webqsp questions. higher formality, readability, and coherency scores for cwq show that the paraphrasing step used by the authors creates more natural and readable questions, as compared to grailqa. we believe that these linguistic differences atleast partially explain the drop in performance for models when tested on cwq. we posit that the higher explicitness of grailqa questions might provide some additional signal to kbqa systems during training that helps them in deciding the best relations/ nodes among the possible options. systems' over-reliance on this signal might not transfer well to other datasets (as shown in this work) thus rendering them less useful. ## conclusion recent kbqa systems have demonstrated impressive performance on the grailqa leaderboard that evaluates them for their zero-shot genearlizability. in this work, we show that these systems that are trained on grailqa do not transfer to other kbqa datasets built on top of the same kb. our analysis shows that despite controlling for structural complexity of the questions, there is a drop in performance across datasets. we observe that this can be explained by the difference in annotation processes and the resulting variations in the linguistic properties of these questions. our work showcases that linguistic variation is an important dimension for evaluating the generalizability of kbqa systems in real-world scenarios.