A New Concept of Knowledge based Question Answering ( KBQA ) System for Multi-hop Reasoning (2022)

Abstract: Knowledge based question answering (KBQA) is a complex task for natural language understanding. Many KBQA approaches have been proposed in recent years, and most of them are trained based on labeled reasoning path. This hinders the system’s performance as many correct reasoning paths are not labeled as ground truth, and thus they cannot be learned. In this paper, we introduce a new concept of KBQA system which can leverage multiple reasoning paths’ information and only requires labeled answer as supervision. We name it as M utliple R easoning P aths KB QA System (MRP-QA). We conduct experiments on several benchmark datasets containing both single-hop simple questions as well as muti-hop complex questions, including WebQuestionSP (WQSP), ComplexWebQuestion-1.1 (CWQ), and PathQuestion-Large (PQL), and demonstrate strong performance.

## introduction knowledge-based question answering (kbqa) is the task of finding answers to questions by processing a structured knowledge base kb. a kb consists of a set of entities e, a set of relations r, and a set of literals s. a knowledge base fact is defined as (h, r, t), where @xmath0, @xmath1, and @xmath2. to answer a simple single-relation question (i.e. a 1-hop question) such as: "who is the president of the united states?", a typical kbqa system first identifies the entity (i.e. united states) and the relation (i.e. "president") asked in the question, and then searches for the answer entity by matching the entity-relation @xmath3, president, ?> over kb. while a single-hop question can be answered by searching a predicate relation in kb, it is much harder to answer more complex multi-hop questions containing multiple entities and relations with constraints. for instance, for complex compositional questions, it is not easy to extract all the relations correctly together with their head and tail entities in the right order. for complex conjunction questions that requires a conjunction of multiple evidences, it is even more difficult to correctly extract all the reasoning paths included. most prior works on multi-hop kbqa focus on learning a single given ground truth reasoning path for each question, and outputting the most possible reasoning path during prediction @xcite @xcite . however, it is common that kb has many alternative paths leading to the correct answer, of various reasoning qualities. these alternative reasoning paths are usually not provided as ground truth by the human annotators. for example, figure 1 shows 7 reasoning paths p @xmath4, . . . , 7}) leading to an answer set containing the correct answer "west lafayette" for a given question "what city is home to the university that is known for purdue boilermakers men's basketball?", but only the reasoning path p 1 is labeled as the correct path in the dataset. a model trained with only p 1 as supervision is likely to miss other paths which are also valid. for example, it will probably map a similar question "what city is home to the stadium that is known for los angeles lakers?" to path p 1 , but fail to associate it with p 3 or p 4 , because p 3 or p 4 contain different types of relations. however, p 1 is a wrong reasoning path for that test question. as the example shown in figure 1 , there are four paths (p 1 ,p 2 ,p 3 ,p 4 ) pointing to the exact answer set containing only the answer entity, and thus can be treated as ground truth paths when training. comparatively, reasoning paths p 5 and p 6 lead to a larger final entity set containing the correct answer each square represents an entity, and the arrows represent the relations. reasoning path p 1 to p 4 are the correct ones containing meaningful reasoning paths to the final answer. p 5 and p 6 are the "second choice" paths that generate a larger final answer set containing some wrong entities. p 7 is the wrong one as its reasoning path is totally not interpretable and the answer set is huge. in this paper, we propose a novel multi-hop kbqa system, which can leverage the training information from multiple reasoning paths (mrps) without using any path annotations, we name it as mutliple reasoning paths kbqa system (mrp-qa). we model the reasoning path as a latent variable, and propose supporting training and prediction methods. the system can output diverse reasoning paths, and reward the "better" paths over the inferior ones by assigning "better" paths higher probabilities. our method can be applied to most kbqa systems to predict the answer, and can be used with any model architecture. we achieve strong performance on three popular kbqa datasets. experimental results show that our model performs especially well on multi-hop question, and in particular on complex questions that cannot be solved with a single reasoning path. our mrp-qa system does not need training paths annotation (only the question, and head and final entities), since it can sample the paths from the kb graph. this is of enormous pratical importance, because in practice questions and answers are easy to collect (sometimes for free), but path annotation is very labor-intensive and expensive. ## model we first introduce some notations. for a given question q and its topic entity e 0 (identified by entity linking tool), a reasoning path is a sequence in the form @xmath5, r 1 , e 1 , r 2 , • • • , e t -1 , r t ) that points to the answer entity e @xmath6. that is, @xmath7. each step (e t-1 , r t , e t ) is a valid fact in the knowledge base kb. our goal is to build a model that can use multiple paths p to predict answer y given question q and topic entity e 0 . in this section, we first present the design of our model architecture, and then explain the training and inference algorithms in detail. ## case study our model requires inference while using the current model to select training samples for next batch in training (see line 6 in algorithm 1). this em style training approach helps us filter out bad reasoning paths based on context information. for example, a sample question from wqsp is who was the owner of kfc?, the graph search algorithm can easily extract two "correct" paths starting from the topic entity kfc directing to the ground truth answer colonel sanders: @xmath8.organization.@xmath9.product.advertising @xmath10. however, the second path is totally wrong given that the reasoning path is irrelevant to the given question. colonel sanders happens to be the advertising character of kfc, but this cannot be generalized to other cases. without using the trained model to filter out this irreverent path, the model may learn incorrect map from who is the owner... to the relation advertising characters. in our experiment, we observe that when we train our model with all reasoning paths generated from dfs algorithm without using this filtering strategy (i.e. k 2 = inf ), the f1 score drops 3.4% as shown in next we demonstrate the benefit of maximizing conditional mutual information instead of likelihood. a sample question in wqsp is who did benjamin franklin get married to?. we observe that there are 13 questions are using benjamin franklin as the topic entity in the training set, but most of them are related to his invention and none of them is about marriage. with such a strong prior on benjamin franklin, our experimental result shows that the model trained with maximum likelihood mistakingly maps this question to a path related to invention, while the model trained with mutual question: what state does romney live in? answer: massachusetts topic entity: romney single ground truth multiple product multiple marginal (our) .89:children .29:education institution/ state province region .83:places lived/ location .06:government positions/ jurisdiction of office .25:places lived/ location .12:government positions/ jurisdiction of office .04:government positions/ office position or title .25:government positions/ district represented .04:government positions/ district represented .00:government positions/ district represented .01:government positions/ jurisdiction of office .01:place of birth/ state .00:place of birth .01:place of birth/ state .00:education/ degree .00:jurisdiction of office .01:sibling/ place of birth .00:election campaigns we further show how generated probabilities look like with different choices of paths and objectives in ## related work most of the existing multi-hop kbqa systems approach this task by decomposing it into two subtasks: topic-entity linking and relation extraction. the topic-entity linking gives the system an entry point to start searching, and the relation extraction is used to search relation paths leading to the final answer. following this track, a straightforward idea is to match the question to a candidate entity/relation directly via calculating the similar-ity between them @xcite @xcite . this method is not ideal for multi-hop questions with long paths, because the number of candidate entity-relation combinations grows exponentially as the number of hops increases. to tackle this issue, methods are proposed to decompose the input question into several single-hop questions, and then use existing method to solve each simple question. the decomposition methods are based on semantic parsing @xcite or templates @xcite . a similar idea is to encode the reasoning information hop by hop, and predict the final answer at the last hop @xcite @xcite . another line of work has looked at solving kbqa task with only final answer as supervision. @xcite first propose to cast kbqa as a program generation task using neural program induction (npi) techniques. they learn to translate the query to a program like logical form executable on the kb. as a follow up, @xcite improves this idea by incorporating high level program structures. both these npi models do not require annotated relation path as supervision, but they need some prior knowledge to design the program templates. in other work, @xcite recently proposed a latent variable approach which is similar to the one described here, but applied on text-based qa scenarios. the main difference between our work is that our method aims at finding multiple reasoning paths leading to the answer, while their method only focus on extracting single optimal solution. we employ inference during training to filter our irrelevant paths, while they use it to identify the optimal solution. besides knowledge based question answering models, there are also other question answering sytstems without using knowledge base/graph in-formation. for examples, there are end-to-end question-answering models by using reinforcement learning @xcite , adversarial learning @xcite , and also the multimodel structure to handle the unknown entities in question answering @xcite . coreference understanding @xcite is also another research direction in designing questionanswering systems. ## conclusion in this paper, we introduce a novel mrp-qa knowledge based question answering system which can leverage information from mrps. to train our model, we use a marginalized probability objective function. experimental results show that our model achieve strong performance on popular kbqa datasets. mantong zhou, minlie huang, and xiaoyan zhu.