Large-Scale Relation Learning for Question Answering over Knowledge Bases with Pre-trained Language Models (2021)

Abstract: The key challenge of question answering over knowledge bases (KBQA) is the inconsistency between the natural language questions and the reasoning paths in the knowledge base (KB). Recent graph-based KBQA methods are good at grasping the topological structure of the graph but often ignore the textual information carried by the nodes and edges. Meanwhile, pre-trained language models learn massive open-world knowledge from the large corpus, but it is in the natural language form and not structured. To bridge the gap between the natural language and the structured KB, we propose three relation learning tasks for BERT-based KBQA, including relation extraction, relation matching, and relation reasoning. By relation-augmented training, the model learns to align the natural language expressions to the relations in the KB as well as reason over the missing connections in the KB. Experiments on WebQSP show that our method consistently outperforms other baselines, especially when the KB is incomplete.

## introduction question answering over knowledge base (kbqa) aims to find the answers to a natural language question given the structured knowledge base (kb) and is widely used in modern question answering and information retrieval systems. traditional retrieval-based kbqa approaches typically build it as a pipeline system, including name entity recognization, entity linking, subgraph retrieval, and entity scoring. in recent years, with the help of deep representation learning, such approaches have achieved remarkable performance @xcite @xcite @xcite @xcite . figure 1 : two error cases from the webqsp @xcite dataset. cvt indicates the compound value type in freebase. for brevity, we abbreviate the name of some entities and relations. however, the kbqa task is still challenging especially for multi-hop questions because of two reasons: 1) due to the complexity of human language, it is often difficult to align the natural language questions with the reasoning paths in the kb. the model tends to learn by surface matching and easily takes shortcut features @xcite for prediction (shown in figure 1a ). 2) in practice, the kb is often incomplete, which also requires the model to reason over the incomplete graph. but the model always fails to do that since it lacks explicit training on reasoning (shown in figure 1b ). previous works such as graftnet @xcite and pullnet @xcite mainly solve these problems by introducing external text corpus (e.g. all wikipedia documents) and use specially designed network architecture to incorporate information from the documents. however, the required external resources may be hard to collect in practice. embedkgqa @xcite solves the kb's incompleteness issue by introducing the pre-trained kb embeddings and trains the ques- question: what was antoni gaudi inspired by ? antoni gaudi william morris visual artist architect influenced_by influenced entity linking path 2: <@xmath0: <@xmath1: <@xmath2: <@xmath3, g) = 0.98 sentence 1: obj{wikipedia} is a registered trademark of the subj{wikimedia foundation, inc.}, a non-profit organization. relation 1: owner of sentence 2: the subj{amagi line} is a japanese railway line connecting kiyama station (on the obj{kagoshima main line}), kiyama and amagi station, asakura. relation 2: connects with relation extraction dataset ↓ [cls] sentence 1 [sep] wikimedia foundation, inc. owner of wikipedia [sep] [cls] sentence 2 [sep] amagi line connects with kagoshima main line [sep] [cls] sentence 1 [sep] sentence 2 [sep] relation extraction (re) task relation matching (rm) task a) the overall architecture of bert-based kbqa approach b) the proposed three auxiliary tasks for relation learning, re, rm and rr. george vi elizabeth ii queen elizabeth the queen mother princess margaret freebase subgraph ↓ relation reasoning (rr) task reasoning path 1 george vi children princess margaret parents queen elizabeth reasoning path 2 george vi children elizabeth ii named_after queen elizabeth target triple george vi people.person.spouse_s queen elizabeth graph linearization [cls] target triple [sep] reasoning path 1 [sep] reasoning path 2 [sep] tion embeddings to be fit in the relation embedding space such that they can directly use the scoring function to rank answers. however, their approach mainly grasps the topological structure of the graph but ignores the textual information in entities and relations that should be also useful to score candidate entities. in this paper, to learn a better mapping from the natural language questions to the reasoning paths in the kb @xcite , we reformulate the retrieval-based kbqa task to make it a question-context matching form and propose three auxiliary tasks for relation learning, namely relation extraction (re), relation matching (rm) and relation reasoning (rr). re and rm both take advantage of the relation extraction datasets, including webred @xcite and fewrel @xcite . re trains the model through inferring relations from the sentences, and rm through determining whether two sentences express the same relation. rr constructs the training data from the kb in a self-supervised manner and trains the model to reason over the missing kb connections given the existing paths. our contributions can be summarized as follows: 1) to bridge the gap between natural language and the structured kb, we reformulate the kbqa task to be a question-context matching problem and propose auxiliary tasks to enhance the implicit relation learning for pre-trained language models @xcite . 2) to mitigate the kb's incompleteness issue, we further propose a task for relation reasoning on the kb. 3) experiments on webqsp show the effectiveness of our proposed approach, especially when the kb is highly incomplete. 1 1 our code is available at https://github.com/ yym6472/kbqarelationlearning 2 approach problem definition in this paper, we mainly focus on the retrieval-based kbqa. given an input query q, we first annotate the named entities in the query and link them to the nodes in the kb. then some heuristic algorithm foot_0 is applied to retrieve a queryspecified subgraph @xmath4, r, e |e, @xmath5, @xmath6, where e is the set of all candidate entities that probably contains the answer of q, and r denotes the relation set. our task is to calculate a score s i for each candidate entity e @xmath7. in this section, we first present how to solve kbqa with bert, then we introduce three proposed auxiliary tasks to augment the relation learning for bert. ## conclusion in this paper, we propose three auxiliary tasks to augment relation learning for bert-based kbqa method, including relation extraction, matching and reasoning. these tasks not only bridge the gap between the natural language and the structured kb, but also explicitly train the model to reason over the incomplete kb. the experimental results on webqsp demonstrate the effectiveness of our approach, especially when the kb is incomplete.