SYGMA : A System for Generalizable and Modular Question Answering Over Knowledge Bases (2022)

Abstract: Knowledge Base Question Answering (KBQA) involving complex reasoning is emerging as an important research direction. However, most KBQA systems struggle with generalizability, particularly on two dimensions: (a) across multiple knowledge bases, where existing KBQA approaches are typically tuned to a single knowledge base, and (b) across multiple reasoning types, where majority of datasets and systems have primarily focused on multi-hop reasoning. In this paper, we present SYGMA, a modular KBQA approach developed with goal of generalization across multiple knowledge bases and multiple reasoning types. To facilitate this, SYGMA is designed as two high level modules: 1) KB-agnostic question understanding module that remain common across KBs, and generates logic representation of the question with high level reasoning constructs that are extensible, and 2) KB-specific question mapping and answering module to address the KB-specific aspects of the answer extraction. We evaluated SYGMA on multiple datasets belonging to distinct knowledge bases (DBpedia and Wikidata) and distinct reasoning types (multi-hop and temporal). State-of-the-art or competitive performances achieved on those datasets demonstrate its generalization capability.

## introduction the goal of knowledge base question answering (kbqa) systems is to answer natural language (nl) questions by retrieving and reasoning over facts in knowledge base (kb). kbqa has gained significant popularity in recent times due to its practical real-world applications and associated research challenges @xcite . however, research in this area has primarily focused so far on single/multi-hop reasoning on a single knowledge base (trivedi et al., 2017; usbeck et al., 2017; † sumit.neelam@in.ibm.com, ‡ udit.sharma@in.ibm.com * this work was done when authors were at ibm. @xcite . as a result, most of the methods developed are tuned to a restricted set of reasoning types on a single knowledge base. in this paper, we present a modular approach for kbqa called sygma (system for generalizable and modular question answering over knowledge bases), that is built on a framework adaptable to multiple kbs and multiple reasoning types. such a system hold promise for many interesting applications namely: answering complex questions involving multiple types of reasoning, unifying of knowledge across multiple kbs, and so on. however, it poses challenges in terms of: (a) handling different representations of information in different kbs. for example, figure 1 shows how information related to question "who was roman emperor before nero?" is represented in two different kbs. wikidata represents properties of facts such as replaces, temporal and spatial with reification foot_0 . on the other hand, dbpedia manages to represent it as a simple fact with a relationship to existing entity nodes. (b) handling a variety of complex reasoning types such as temporal and spatial in a common framework. in sygma, we tackle these challenges through: (1) a modular design of the system and (2) using λcalculus based intermediate representations. modular design offers flexibility to isolate sub-tasks that need adaptation, thus avoiding data-intensive endto-end adaptation. motivated by nsqa @xcite and retrack @xcite , ## sygma: system description figure 2 shows overall architecture of sygma. motivated by @xcite , sygma is designed as a modular system, where multiple submodules performing distinct sub-tasks are stitched together in a pipeline. each of these sub-modules are built independent of each other with sub-task specific data. such a design offers greater flexibility to push ahead our goal of generalization, because it is possible to isolate those sub-tasks that need adaptation and restrict adaptation effort only to the corresponding sub-modules, unlike data-intensive end-to-end trained systems. kb generalization: towards the goal of kb generalization, the sub-modules are grouped into two processing stages, as shown in figure 2 : 1) kb-agnostic question understanding that transforms natural language (nl) question into a kb-agnostic logical representation of the question, and 2) kb-specific question mapping and reasoning that maps the elements of the kb-agnostic logical representation onto the vocabulary of the kb to first build a kb-specific logical representation of the question and then transform that further into a sparql query that when executed over the kb would fetch the answer. figure 3 gives an illustration of the outputs generated at different intermediate stages of the pipeline for an example nl question. note that, in our effort to achieve generalization across kbs, it is not possible to avoid the kb-specific components completely out from the system. through above design, all we try to achieve is to minimize the adaptation effort by pushing all kb-specific processing towards the end of the pipeline, so the preceding part of the pipeline would remain common across all the kbs and the efforts needed to support new kbs is limited only to the later part of the pipeline. reasoning type generalization: support for generalization across reasoning types is achieved through: (1) use of λ-calculus for logical representation that captures different reasoning types as higher order functions, and (2) incorporation of transformation heuristics within logical representation modules that spans across both the stages of the pipeline, as shown in figure 2 . λ-calculus provides a flexible framework not only to handle a variety of reasoning types but also to handle representation differences across kbs. over that, the transformation modules incorporate necessary high-level knowledge about different reasoning constructs and their transformations that are needed to handle specifics of the different reasoning types. note that we chose to adopt such a neuro-symbolic approach of combining acquired knowledge with the learned knowledge in a modular fashion. on this aspect, we would like to highlight that end-to-end trained systems do not have any distinctive advantage, because to acquire such a knowledge through data is difficult and would need large amount of manual effort in terms of data collection and labeling, to cover a wide range of reasoning types. before we get into the details of different modules, next we describe lambda calculus, the formalism used in our system for logical representation. ## results & discussion note that sygma is the only system able run on all the datasets, which indeed demonstrate the main goal of our work. although it is not able to achieve top performance on dbpedia datasets, generalizability across kbs and reasoning types with performance comparable to state-of-the-art is the key distinguishing aspect of sygma. ## conclusion in this paper, we described sygma, a kbqa system that generalize across kbs and reasoning types. the key distinguishing features of our approach are its modular design and its use of amr/λ-calculus based question understanding and decomposition module to obtain kb-agnostic logical representation of the question. it also includes kb-specific question mapping and reasoning modules and transformation heuristics to handle multiple kbs and multiple reasoning types. an experimental evaluation of sygma on datasets spanning multiple kbs and reasoning types indeed demonstrates its generalization capabilities. among many potential opportunities for future work, we believe improving the individual performance of relation linking module is critical and holds promise for significantly improving the overall performance.