Retrieval, Re-ranking and Multi-task Learning for Knowledge-Base Question Answering (2021)

Abstract: Question answering over knowledge bases (KBQA) usually involves three sub-tasks, namely topic entity detection, entity linking and relation detection. Due to the large number of entities and relations inside knowledge bases (KB), previous work usually utilized sophisticated rules to narrow down the search space and managed only a subset of KBs in memory. In this work, we leverage a retrieve-and-rerank framework to access KBs via traditional information retrieval (IR) method, and re-rank retrieved candidates with more powerful neural networks such as the pre-trained BERT model. Considering the fact that directly assigning a different BERT model for each sub-task may incur prohibitive costs, we propose to share a BERT encoder across all three sub-tasks and define task-specific layers on top of the shared layer. The unified model is then trained under a multi-task learning framework. Experiments show that: (1) Our IR-based retrieval method is able to collect high-quality candidates efficiently, thus enables our method adapt to large-scale KBs easily; (2) the BERT model improves the accuracy across all three sub-tasks; and (3) benefiting from multi-task learning, the unified model obtains further improvements with only 1/3 of the original parameters. Our final model achieves competitive results on the SimpleQuestions dataset and superior performance on the FreebaseQA dataset.

## introduction answering natural language questions by searching over large-scale knowledge bases (kbqa) is highly demanded by real-life applications, such as google assistant, siri, and alexa. owing to the availability of large-scale kbs, significant advancements have been made over the years. one main research direction views kbqa as a semantic matching task @xcite @xcite @xcite @xcite @xcite , and finds a relation-chain within kbs that is most similar to the question in a common semantic space, where the relation-chain can be 1-hop, 2-hop or multi-hop @xcite . another research direction formulates kbqa as a semantic parsing task @xcite @xcite , and tackles questions that involve complex reasoning, such as ordinal (e.g. what is the second largest fulfillment center of amazon?), and aggregation (e.g. how many fulfillment centers does amazon have?). most recently, some studies proposed to derive answers from both kbs and free-text corpus to deal with the low-coverage issue of kbs @xcite @xcite . in this paper, we follow the first research direction since the relationchain type of questions counts the vast majority of real-life questions @xcite @xcite . previous semantic matching methods for kbqa usually decompose the task into sequential subtasks consisting of topic entity detection, entity linking, and relation detection. for example in figure 1 , given the question "who wrote the book beau geste?", a kbqa system first identifies the topic entity "beau geste" from the question, then the topic entity is linked to an entity node (m.04wxy8) from a list of candidate nodes, and finally the relation book.written work.author is selected as the relation-chain leading to the final answer. previous methods usually worked on a subset of kb in order to fit kb into memory. for entity linking, some sophisticated heuristics were commonly used to collect entity candidates. for relation detection, previous work usually enumerated all possible 1-hop and 2-hop relation-chains (starting from linked entity nodes) as candidates. all who wrote the book beau geste ? mid name type m.04wxy8 beau geste book m.0dl_h4 beau geste film m.051vvdc beau geste music … … … (2) entity linking predicate object mid book.written_work.author m.05f834 book.written_work.subjects m.0322m book.book.genre m.05hgj … … (3) relation detection (1) topic entity detection node candidates for entity linking. relation-chain candidates for relation detection. these workarounds may prevent their methods from generalizing well to other datasets, and scaling up to bigger kbs. to tackle these issues, we leverage a retrieveand-rerank strategy to access kbs. in the retrieval step, we ingest kbs into two inverted indices: one that stores all entity nodes for entity linking, and the other one that stores all subject-predicate-object triples for relation detection. then, we use tf-idf algorithm to retrieve candidates for both entity linking and relation detection sub-tasks. this method naturally overcomes the memory overhead when dealing with large-scale kbs, therefore makes our method easily scale up to large-scale tasks. in the re-ranking step, we leverage the advanced bert model to re-rank all candidates by fine-grained semantic matching. for the topic entity detection sub-task, we utilize another bert model to predict the start and end positions of a topic entity within a question. since assigning a different bert model for each sub-task may incur prohibitive costs, we therefore propose to share a bert encoder across sub-tasks and define task-specific layers for each individual sub-task on top of the shared layer. this unified bert model is then trained under the multitask learning framework. experiments on two standard benchmarks show that: (1) our ir-based retrieval method is able to collect high-quality candidates efficiently; (2) the bert model improves the accuracy across all three sub-tasks; and (3) benefiting from multi-task learning, the unified model obtains further improvements with only 1/3 of the original parameters. our final model achieves competitive results on the simplequestions dataset and superior performance on the freebaseqa dataset. ## task definition knowledge-base question answering (kbqa) aims to find answers for natural language questions from structural knowledge bases (kb). we assume a kb k is a collection of subject-predicate-object triples e 1 , p, e 2 , where e 1 , e 2 ∈ e are entities, and @xmath0, e is the set of all entities, and p is the set of all relation types. given a question q, the goal of kbqa is to find an entity node @xmath1, thus can be formulated as ## background we briefly introduce some background required by the following sections. bert: bert model @xcite follows the multi-head self-attention architecture @xcite , and is pre-trained with a masked language modeling objective on a largescale text corpus. it has achieved state-of-the-art performance on a bunch of textual tasks. specifically, for semantic matching tasks, bert simply concatenates two textual sequences together, and encodes the new sequence with multiple selfattention layers. then, the output vector of the first token is fed into a linear layer to compute the similarity score between the two input textual sequences. freebase: we take freebase @xcite as our back-end kb to answer questions. it contains more than 46 million topic entities and 2.6 billion triples. each entity has an internal machine identifier (mid) and a set of aliases. some entities also have properties such as entity types and detailed descriptions. freebase contains a special entity category called compound value type (cvt), which does not have a name or alias, and is only used to collect multiple fields of an event or a special relationship. in the official freebase dump foot_0 , all facts are formulated as the unified subjectpredicate-object triples, and there is no explicit split for entities and relations. we partition facts in freebase into a set of entities e and a set of relations k by following the pre-processing steps in @xcite . inverted index and tf-idf: inverted index is an optimized data structure of finding documents (from a large document collection) where a query word x occurs. it is commonly used for fast freetext searches. term frequency-inverse document 4 retrieval and re-ranking for kbqa in this section, we describe how to parameterize p t , p l and p r in equation (2). ## experiments we evaluate the effectiveness of our model on standard benchmarks in this section. we first conduct experiments on each sub-task with a separate bert model in section 6.2, 6.3 and 6.4, then evaluate the influence of sharing a bert encoder for all three models in section 6.5. finally, we benchmark our method on full freebase in section 6.6. ## conclusion in this work, we proposed a retrieve-and-rerank strategy to access large-scale kbs in two steps. first, we leveraged traditional ir methods to collect high-quality candidates from kbs for entity linking and relation detection. second, we utilized the advanced bert model to re-rank candidates by fine-grained semantic matching. we also employed a bert model to predict the start and end positions of the topic entity in a question. to reduce the model size, we proposed a joint model to share bert encoder across all three sub-tasks, and create task-specific layers on the top. we then trained this joint model with multi-task learning. experimental results show that our method achieves superior results on standard benchmarks, and is able to scale up to large-scale kbs.