Relation-Guided Pre-Training for Open-Domain Question Answering (2021)

Abstract: Answering complex open-domain questions requires understanding the latent relations between involving entities. However, we found that the existing QA datasets are extremely imbalanced in some types of relations, which hurts the generalization performance over questions with long-tail relations. To remedy this problem, in this paper, we propose a Relation-Guided Pre-Training (RGPT-QA) framework. We first generate a relational QA dataset covering a wide range of relations from both the Wikidata triplets and Wikipedia hyperlinks. We then pre-train a QA model to infer the latent relations from the question, and then conduct extractive QA to get the target answer entity. We demonstrate that by pre-training with propoed RGPT-QA techique, the popular open-domain QA model, Dense Passage Retriever (DPR), achieves 2.2%, 2.4%, and 6.3% absolute improvement in Exact Match accuracy on Natural Questions, TriviaQA, and WebQuestions. Particularly, we show that RGPT-QA improves significantly on questions with long-tail relations.

## introduction open domain question answering is a challenging task that answers factoid questions based on evidence in a large corpus (e.g., wikipedia). most open-domain qa systems follow retriever-reader pipeline @xcite , in which a retriever selects a subset of candidate entities and associated passages from the corpus that might contain the answer, then a reader extracts a text span from the passages as the answer. this process involves multiple entities that are relevant to answer the question. the qa system is required to extract these entities from the question and passages and identify the (latent) semantic relations between these entities in order to answer the question. for example, to answer the following question: "where did steph curry play college basketball at?", the qa model is required to reason the implicit relation triplet steph curry, educated at, davidson college to identify the correct answer. to capture the relation knowledge required to answer questions, most qa systems rely on humanannotated supervised qa datasets. however, it is expensive and tedious to annotate a large set of qa pairs that cover enough relational facts for training a strong qa model. in addition, we showed that even for a large qa dataset like natural questions @xcite , its training set only covers 16.4% of relations in wikidata (vrandecic and krötzsch, 2014) knowledge graph. moreover, for those covered relations, the frequency distribution is imbalanced, i.e., 30% of relation types appear only once. consequently, for the questions involving infrequent (a.k.a, long-tail) relations in the training set, the qa exact match accuracy is 22.4% lower than average. such a biased relation distribution of existing qa datasets severely hurts the generalization of trained qa systems. to improve the open-domain qa systems for questions with long-tail relations, in this paper, we propose rgpt-qa, a simple yet effective relation-guided pre-training framework for training qa models with augmented relationa facts from knowledge graph. the framework consists of two steps: 1) generate a relational qa dataset that covers a wide range of relations without human labeling; 2) pre-train a qa model to predict latent relations from questions and conduct extractive qa. the key of our framework is to generate a relational qa dataset that align entities in wikipedia passages with structured knowledge graph (e.g., wikidata). we call such a dataset grounded relational wiki-graph. in this graph, each edge indicates the relationship of two connected entities, and the edge is linked to a passage in wikipedia describing this relationship. as wikidata knowledge graph also suffers from low coverage of long-tail entities and relations, we further convert hyperlinks in wikipedia into knowledge triplets without specifying relation labels. next, we link each relation triplet to a wikipedia passage to help generate natural questions. we assume that if one passage in the wiki-page of source entity contains the target entity, then the context in this passage describes the relationship between the two entities. with the constructed graph, we use a template to synthesize question and answer pairs and then pre-train the qa model to capture the relational facts for answering complex open-domain questions. as a pre-training method, rgpt-qa can be incorporated with any open-domain qa system. in this paper, we utilize the recently developed dense passage retriever (dpr) @xcite as the base qa system to evaluate the proposed pretraining effectiveness. experimental results show that rgpt-qa enhances dpr's exact match accuracy by 2.2%, 2.4%, and 6.3% on natural questions, triviaqa and webquestions respectively. compared with the existing qa pre-training methods @xcite @xcite , rgpt-qa explicitly captures a wide range of relational facts and thus achieves better performance. moreover, for the questions containing long-tail relations in natural questions, the performance is improved by 10.9%, showing that rgpt-qa alleviates the unbalanced relation distribution problem in the existing qa datasets. the key contributions of this paper are: ## preliminary and empirical analysis in this section, we firstly introduce the retrieverreader pipeline for open-domain qa, and then we analyze how the relation distribution in existing qa datsets influence generalization performance. ## method in this section, we will discuss rgpt-qa framework in: 1) how to generate relational qa dataset for the pre-training purpose; and 2) how to construct a self-training task to empower qa model to capture relational facts. ## experiments in this section, we evaluate rgpt-qa on three open-domain qa datasets: natural questions (nq), trivia qa and web questions (wq). ## conclusion in this paper, we propose a simple yet effective pre-training framework rgpt-qa. we leverage both the wikipedia hyperlinks and wikidata relation triplets to construct grounded relational wiki-graph, based on which we generate relational qa dataset. we then pre-train a qa model to infer the latent relation from the question, and then conduct extractive qa to get the target answer entity. rgpt-qa improves the performance of the stateof-the-art qa frameworks, especially for questions with long-tail relations.