Answering Complex Questions by Combining Information from Curated and Extracted Knowledge Bases (2020)

Abstract: Knowledge-based question answering (KB_QA) has long focused on simple questions that can be answered from a single knowledge source, a manually curated or an automatically extracted KB. In this work, we look at answering complex questions which often require combining information from multiple sources. We present a novel KB-QA system, Multique, which can map a complex question to a complex query pattern using a sequence of simple queries each targeted at a specific KB. It finds simple queries using a neural-network based model capable of collective inference over textual relations in extracted KB and ontological relations in curated KB. Experiments show that our proposed system outperforms previous KB-QA systems on benchmark datasets, ComplexWebQuestions and WebQuestionsSP.

## introduction knowledge-based question answering (kb-qa) computes answers to natural language questions based on a knowledge base. some systems use a curated kb @xcite , and others use an extracted kb @xcite . the choice of the kb depends on its coverage and knowledge representation: a curated kb uses ontological relations but has limited coverage, while an extracted kb offers broad coverage with textual relations. commonly, a kb-qa system finds answers by mapping the question to a structured query over the kb. for instance, example question 1 in fig. 1 can be answered with a query (rihanna, place of birth, ?) over a curated kb or (rihanna, 'was born in', ?) over an extracted kb. most existing systems focus on simple questions answerable with a single kb. limited efforts have been spent to support complex questions that 2. what college did the author of 'the hobbit' attend? require inference over multiple relations and entities. for instance, to answer question 2 in fig. 1 , we need to infer relations corresponding to expressions 'author of' and 'attend'. in practice, a single kb alone may not provide both high coverage and ontological knowledge to answer such questions. a curated kb might provide information about educational institutions, while an extracted kb might contain information about authorship. leveraging multiple kbs to answer complex questions is an attractive approach but is seldom studied. existing methods assume a simple abstraction @xcite over the kbs and have limited ability to aggregate facts across kbs. we aim to integrate inference over curated and extracted kbs for answering complex questions. combining information from multiple sources offers two benefits: evidence scattered across multiple kbs can be aggregated, and evidence from different kbs can be used to complement each other. for instance, inference over ontological relation book author can benefit from textual relation 'is written by'. on the other hand, evidence matching 'attend' may exclusively be in the curated kb. example 1 what college did the author of 'the hobbit' attend? simple queries: g 1 : the hobbit 'is wrtten by' ?a. g 2 : ?b person.education ?c . ?c institution ?x. our goal is to map a complex question q to a query g, which can be executed against a combination of curated kb k c and extracted kb k o . knowledge bases. the background knowledge source @xmath0, k o } is denoted as @xmath1, e, r), where v is the set of entities and e is a set of triples (s, r, o). a triple denotes a relation @xmath2. the relation set r is a collection of ontological relations r o from k c and textual relations r t from k o . a higher order relation is expressed using multiple triples connected using a special cvt node. complex question, q corresponds to a query g which has more than one relation and a single query focus ?x. g is a sequence of partial queries @xmath3, g 2 , .., g o ) connected via different join conditions. a partial query has four basic elements: a seed entity s r is the root of the query, a variable node o v corresponds to an answer to the query, a main relation path (s r , p, o v ) is the path that links s r to o v by one or two edges from either r o or r t , and constraints take the form of an entity linked to the main relation by a relation c. by definition, each partial query targets a specific kb. a composition tree c describes how the query g is constructed and evaluated given the partial queries. it includes two functions, simqa and join. simqa is the model for finding simple queries. it enumerates candidates for a simple query, encodes and compares them with the question representation, and evaluates the best candidate. join describes how to join two partial queries i.e. whether they share the query focus or another variable node. fig. 2 shows the partial queries and composition tree for the running example 1. overview. given a complex input question, the task is to first compute a composition tree that describes how to break down the inference into simple partial queries. we then have to gather can- ## partial query candidate generation we first describe how we find candidates for partial queries given an input question. we use a staged generation method with staged states and actions. compared to previous methods @xcite which assume a question has one main relation, our strategy can handle complex questions which have multiple main relations (and hence partial queries). we include a new action a t that denotes the end of the search for a partial query and transition to a state s t . state s t refers back to the composition tree to determine the join condition between the current partial query and the next query. if they share an answer node, candidate generation for the subsequent query can resume independently. otherwise, it waits for the answers to the current query. we generate (entity, mention) pairs for a question using entity linking @xcite and then find elements for query candidates. fig. 4 depicts our staged generation process. identify seed entity. the seed s r for a partial query is a linked entity in the question or an answer of a previously evaluated partial query. identify main relation path. given a seed entity, we consider all 1-hop and 2-hop paths p. these include both ontological and textual relations. the other end of the path is the variable node o v . ## semantic matching we now describe our neural-network based model which infers over different relation forms and computes the semantic similarity of a partial query candidate to the question. ## query composition in this work, we focus on constructing complex queries using a sequence of simple partial queries, each with one main relation path. since the original question does not have to be chunked into simple questions, constructing composition trees for such questions is fairly simple. heuristically, a composition tree can simply be derived by estimating the number of main relations (verb phrases) in the question and the dependency between them (subordinating or coordinating). we use a more sophisticated model @xcite to derive the composition tree. the post-order traversal of the tree yields the order in which partial queries should be executed. given a computation tree, we adopt a beam search and evaluate best k candidates for a partial query at each level. this helps maintain tractability in the large space of possible complex query derivations. the semantic matching model only independently scores the partial queries and not complete derivations. we, thus, need to find the best derivation that captures the meaning of the complex input question. to determine the best derivation, we aggregate the scores over the partial queries and consider additional features such as entities and structure of the query. we train a log-linear model on a set of (question-answer) pairs using features such as semantic similarity scores, entity linking scores, number of constraints in the query, number of variables, number of relations and number of answer entities. given the best scoring derivation, we translate it to a kb query and evaluate it to return answers to the ques-tion. such an approach has been shown to be successful in answering complex questions over a single knowledge base @xcite . in this work, we extend that approach to scenarios when multiple kbs are available. ## experiments we present experiments that show multique outperforms existing kb-qa systems on complex questions. our approach to construct queries from simple queries and aggregate multiple kbs is superior to methods which map questions directly to queries and use raw text instead.