RNG - KBQA : Generation Augmented Iterative Ranking for Knowledge Base Question Answering (2022)

Abstract: Existing KBQA approaches, despite achieving strong performance on i.i.d. test data, often struggle in generalizing to questions involving unseen KB schema items. Prior ranking-based approaches have shown some success in generalization, but suffer from the coverage issue. We present RnG-KBQA, a Rank-and-Generate approach for KBQA, which remedies the coverage issue with a generation model while preserving a strong generalization capability. Our approach first uses a contrastive ranker to rank a set of candidate logical forms obtained by searching over the knowledge graph. It then introduces a tailored generation model conditioned on the question and the top-ranked candidates to compose the final logical form. We achieve new state-of-the-art results on GrailQA and WebQSP datasets. In particular, our method surpasses the prior state-of-the-art by a large margin on the GrailQA leaderboard. In addition, RnG-KBQA outperforms all prior approaches on the popular WebQSP benchmark, even including the ones that use the oracle entity linking. The experimental results demonstrate the effectiveness of the interplay between ranking and generation, which leads to the superior performance of our proposed approach across all settings with especially strong improvements in zero-shot generalization.

## introduction modern knowledge bases (kb) are reliable sources of a huge amount of world knowledge but can be difficult to interact with since they are extremely large in scale and require specific query languages (e.g., sparql) to access. question answering over knowledge base (kbqa) serves as a user-friendly way to query over kbs and has garnered increasing attention @xcite . recent research has attempted to build systems given a question, we first rank logical form candidates obtained by searching over the kb based on predefined rules. here, the ground truth logical form is not in the top-ranked candidates as it is not covered by the rules. we solve this problem using another generation step that produces the correct logical form based on topranked candidates. the final logical form is executed over the kb to yield the answer. we propose rng-kbqa, a new framework targeted at generalization problems in the task of kbqa. our approach combines a ranker with a generator, which addresses the coverage issue in ranking-only based approaches while still benefiting from their generalization power. as shown in figure 1 , we first employ a ranker to select a set of related logical forms from a pool of candidate logical forms obtained by searching over the graph. the selected logical forms are not required to cover the correct one, but are semantically coherent and aligned with the underlying intents in the question. next, we introduce a generator that consumes both the question and the top-k ranked candidates to compose the final logical form. the core idea of our approach is the interplay between the ranker and the generator: the ranker provides essential ingredients of kb schema items to the generator, which then further refines the top-candidates by complementing missing constructions or constraints, and hence allows covering a broader range of logical form space. we base both our ranker and generator on pretrained language models for better generalization capability. unlike prior systems which rank candidates using a grammar-based parser @xcite or a seq-to-seq parser @xcite , our ranker is a bert-based @xcite biencoder (taking as input question-candidate pair) trained to maximize the scores of ground truth logical forms while minimizing the scores of incorrect candidates. such training schema allows learning from the contrast between the candidates in the entire territory, whereas prior parsing-based ranker @xcite only learns to encourage the likelihood of the ground truth logical forms. we further develop an iterative-bootstrapbased training curriculum for efficiently training the ranker to distinguish spurious candidates (section 2.2). in addition, we extend the proposed logical form ranker, keeping the architecture and logic the same, for the task of entity disambiguation, and show its effectiveness as a second-stage entity ranker. our generator is a t5-based @xcite seq-to-seq model that fuses semantic and structural ingredients found in top-k candidates to compose the final logical form. to achieve this, we feed the generator with the question followed by a linearized sequence of the top-k candidates, which allows it to distill a refined logical form that will fully reflect the question intent by complementing the missing pieces or discarding the irrelevant parts without having to learn the low-level dynamics. we test rng-kbqa on two datasets, grailqa and webqsp, and compare against an array of strong baselines. on grailqa, a challenging dataset focused on generalization in kbqa, our approach sets the new state-of-the-art performance of 68.8 exact match 74.4 f1 score, surpassing prior sota (58.1 exact match and 65.3 f1 score) by a large margin. on the popular webqsp dataset, rng-kbqa also outperforms the best prior approach (qgg @xcite ) and achieves a new sota performance of 75.7 f1 score. the results demonstrate the effectiveness of our approach across all settings and especially in compositional generalization and zero-shot generalization. ## experiments we mainly test our approach on grailqa @xcite , a challenging dataset focused on evaluating the generalization capabilities. we also ex-overall i.i.d. compositional zero-shot em f1 em f1 em f1 em f1 qgg (lan and jiang, 2020) -36.7 -40.5 -33.0 -36.6 ## related work kbqa is a promising technique for users to efficiently query over large kb, which has been extensively studied over the last decade. past work has collected a series of datasets @xcite @xcite @xcite as well as proposed a diversity of approaches for this task. one line of kbqa approaches first constructs a query-specific subgraph with information retrieved from the kb and then rank entity nodes to select top entities as the answer @xcite @xcite @xcite . the subgraph can either be retrieved in one-shot using heuristic rules @xcite , or iteratively built using learned models @xcite @xcite . later, a neural model operating over subgraph is employed to determine the answer nodes @xcite @xcite . such information retrieval based approaches are usually less interpretable as they do not produce the inference path reaching the answer, whereas our approach is more transparent since we are able to produce logical forms. more closely related to our approach, another line answers a question by parsing it into an executable logical form in various representations, including lambda-dcs @xcite , sparql query @xcite , graph query @xcite @xcite , and s-expression @xcite . past work has attempted to generate logical forms using grammar-based parsera @xcite or seq-to-seq parsers @xcite . there has also been an alternative way that first enumerates a list of logical form candidates and then choose one that best matches the intents in the question @xcite @xcite @xcite @xcite . our approach differs in that we employ a generation stage to remedy the coverage issue which these approaches often suffer from. ## conclusion we have presented rng-kbqa for question answering over knowledge base. rng-kbqa consists of a ranking step and a generation step. our ranker trained with iterative bootstrapping strategy can better distinguish correct logical forms from spurious ones than prior seq-to-seq ranker. our generator can further remedy uncovered operations or implicitly mentioned constraints in the top-ranked logical forms. the experimental results on two datasets, grailqa and webqsp, suggest the strong performance of our approach: rng-kbqa achieves new state-of-the-art performance on both datasets, and particularly outperforms prior methods in generalization setting by a large margin.