A Two-Stage Approach towards Generalization in Knowledge Base Question Answering (2022)

Abstract: Most existing approaches for Knowledge Base Question Answering (KBQA) focus on a specific underlying knowledge base either because of inherent assumptions in the approach, or because evaluating it on a different knowledge base requires non-trivial changes. However, many popular knowledge bases share similarities in their underlying schemas that can be leveraged to facilitate generalization across knowledge bases. To achieve this generalization, we introduce a KBQA framework based on a 2-stage architecture that explicitly separates semantic parsing from the knowledge base interaction, facilitating transfer learning across datasets and knowledge graphs. We show that pretraining on datasets with a different underlying knowledge base can nevertheless provide significant performance gains and reduce sample complexity. Our approach achieves comparable or state-of-the-art performance for LC-QuAD (DBpedia), WebQSP (Freebase), SimpleQuestions (Wikidata) and MetaQA (Wikimovies-KG).

## introduction knowledge base question answering (kbqa) has gained significant popularity in recent times due to its real-world applications, facilitating access to rich knowledge graphs (kgs) without the need for technical query-syntax. given a natural language question, a kbqa system is required to find an answer based on the facts available in the kg. for example, given the question "who is the director of the film titanic", a kbqa system should retrieve the entity corresponding to "james cameron". this would be dbr:james _ cameron foot_0 in dbpedia @xcite , wd:q42574 foot_1 in wikidata @xcite , and fb:m.03 _ gd foot_2 in freebase @xcite . kbqa has been evaluated on multiple different kgs such as freebase @xcite , wikidata @xcite , dbpedia @xcite , and metaqa @xcite . most existing heuristic-based kbqa approaches such as nsqa @xcite , ganswer @xcite , and qamp @xcite are typically tuned for a specific underlying knowledge base making it non-trivial to generalize to other kgs. on the other hand, wdaqua @xcite , a system with a focus on being generalizable, ignores question syntax, thereby showing reduced performance on datasets with complex multi-hop questions. recently, there has been a surge in end-to-end learning approaches that are not tied to specific kgs or heuristics, and hence can generalize to multiple kgs. grailqa @xcite in particular categorized different forms of generalization, such as novel relation compositionality and zero-shot generalization. they also demonstrated transfer across qa datasets, but within the same kg. on the other hand, graftnet @xcite and em-bedkgqa @xcite demonstrated their ability to generalize over multiple kgs with stateof-the-art performance on metaqa (wikimovies) as well as webqsp (freebase). the two techniques, however, are highly sensitive to the training data; failing to generalize in terms of relation compositionality within a kg. their performance drops (between 23-50%) on relation compositions that are not seen during training. in this work, we present a novel generalizable kbqa approach stag-qa (semantic parsing for transfer and generalization) that works seamlessly across multiple kgs. we demonstrate this on kbqa datasets with different underlying kgs. our approach attempts to separate aspects of kbqa systems that are softly tied to the kg but generalizable, from the parts more strongly tied to a specific kg. concretely, our approach has two stages: 1) the first stage is a generative model that predicts a query skeleton, which includes the query pattern, the different sparql operators in it, as well as partial relations based on label semantics that can be generic to most knowledge graphs. 2) the second stage converts the output of the first stage to a final query that includes entity and relations mapped to a specific kg to retrieve the final answer. our contributions are as follows: ## proposed architecture the kbqa task involves finding an answer for a natural language question from a given kg. following the semantic parsing techniques for kbqa @xcite @xcite , we attempt to solve this task by predicting the correct structured sparql query that can retrieve the required answer(s) from the kg, i.e, by estimating a probability distribution over possible sparql queries given the natural language question. in this work, we aim to design a model architecture that generalises across different kgs such as dbpedia, wikidata, and freebase. in order to achieve this goal, we have a two-stage approach as shown in figure 1 , where we separate generic sparql query-sketch learning from kg-specific mapping of concepts. specifically, the following two-stages are: softly-tied query sketch: this is the first stage of our approach where we learn aspects of the sparql query generation that are generic to any kg. specifically, we observe the following: (i) multi-hop patterns are mostly generic to question answering over kgs. (ii) across many kgs, analogous relations have semantic or lexical overlap. therefore, we focus on two sub-tasks in this stage, query skeleton generation and partial relation linking. we call the output of this stage a softly-tied semantic parse, because the exact output is partially dependent on the specific kg in use, but our choice of representations and architecture ensures that transfer across kgs is a natural consequence. kg alignment: here we introduce all vocabulary specific to the kg in order to generate an executable sparql query. to do so, we bind the softy-tied semantic parse strongly to the kg to find the answer by (i) resolving the textual relations to kg relations, (ii) introducing kg specific entities into the sparql skeleton, and (iii) rank the obtained sparql queries based on its groundings in the kg. ## experiments in this section, we validate two claims: (1) stag-qa achieves state-of-the-art or comparable performance on a variety of datasets and kgs. (2) stag-qa generalizes across kbs and hence facilitating transfer. all experiments were run on a machine with an intel xeon e5-2690 cpu and 1 x p100 gpu. sparql virtuoso endpoints were setup on the local network for all kbs. runtime was approximately 2.28 seconds per question. the results show that pre-training our system even on a different kg achieves improvement in performance with better gains in low-resource and unseen relation combination settings. ## related work there have been a variety of kbqa systems developed recently. closest among them to our architecture is masp, a multi-task end-to-end learning approach that focuses of dialog-based qa. however, masp does not show transfer across kgs. coarse2fine @xcite is another two-stage approach, but they differ in the completeness of the semantic parse, while the two stages in our approach differ in kb-agnosticity. some kbqa approaches are rule based, and depend on generic language based syntactic @xcite or semantic parses @xcite of the question and build rules on it to obtain a query graph that represents the sparql query. nsqa, the state of the art approach for dbpedia based datasets such as lc-quad-1.0 @xcite and qald-9 @xcite , falls in this category. many of these systems have components that are specific to the kg they evaluate on, and do not trivially generalize to other kgs. in particular ganswer, nsqa, and qamp are specific to dbpedia and do not evaluate their approaches on any other kgs. another class of approaches explicitly generate query-graphs through a staged generation procedure @xcite . the primary difference between these works and ours is that they fully resolve the entities and relations in each stage of the query graph construction process; thus the graph is strongly tied to the kg at each stage. on the other hand, we separate the structure of the graph (skeleton) from the actual entities and relations. transfer learning between kgs has not been empirically validated for these methods. a prominent work @xcite ) ranks all candidate graph patterns retrieved from the knowledge graph based on the grounded entity. in multi-hop settings, as in metaqa with 3-hop questions, retrieving all possible candidates upto n-hops (for an arbitrary choice of n) and then ranking across all of them is expensive. in contrast, our work focuses on a generative approach to model query graph patterns. embedkgqa @xcite and graftnet are two approaches that directly ranks across entities in the knowledge base to predict an answer, by leveraging either kg embeddings from knowledge base completion (kbc); or creating a unified graph from kb and text. however, these approaches do not generalize well to novel relation compositions not seen during training. finally, it is unclear how to transfer kbc embedding-based approaches such as embed-kgqa across kgs since the learnt kg embeddings are tightly coupled with the kg. ## conclusion in this work, we show that a simple 2-stage architecture which explicitly separates the kg-agnostic semantic parsing stage from the kg-specific interaction can generalize across a range of datasets and kgs. we evaluated our approach on four kg/qa pairs, obtaining state-of-the-art performance on metaqa, lc-quad 1.0, and simplequestions-wiki; as well as competitive performance on we-bqsp. we also successfully demonstrate transfer learning across kgs by showing that pre-training the first stage on an existing kg/qa-dataset pair can help improve performance in low-resource settings for a new target kg; and greatly reduce the sample complexity to achieve state-of-the-art performance. finally, we show that some popular benchmark datasets do not evaluate generalization to novel relation compositionality.