Investigating the Effect of Background Knowledge on Natural Questions (2021)

Abstract: Existing work shows the benefits of integrating KBs with textual evidence for QA only on questions that are answerable by KBs alone (Sun et al., 2019). In contrast, real world QA systems often have to deal with questions that might not be directly answerable by KBs. Here, we investigate the effect of integrating background knowledge from KBs for the Natural Questions (NQ) task. We create a subset of the NQ data, Factual Questions (FQ), where the questions have evidence in the KB in the form of paths that link question entities to answer entities but still must be answered using text, to facilitate further research into KB integration methods. We propose and analyze a simple, model-agnostic approach for incorporating KB paths into text-based QA systems and establish a strong upper bound on FQ for our method using an oracle retriever. We show that several variants of Personalized PageRank based fact retrievers lead to a low recall of answer entities and consequently fail to improve QA performance. Our results suggest that fact retrieval is a bottleneck for integrating KBs into real world QA datasets

## introduction prior work has shown the benefit of retrieving paths of related entities @xcite @xcite and learning relevant knowledge graph embeddings @xcite @xcite for answering questions on kbqa datasets such as webquestions @xcite and metaqa @xcite . but such datasets are often curated to questions with kb paths that contain the right path to the answer and hence are directly answerable via kb. an open question remains whether such approaches are useful for questions not specifically designed to be answerable by kbs. in this paper, we aim to evaluate kb integration for real-world qa settings in the context of the natural questions (nq) dataset @xcite which consists of questions naturally posed by users of a search engine. nq is one of the common benchmarks that is used to test the real-world qa applicability of models, hence motivating our choice. to study the effect of augmenting kb knowledge, we construct a subset of nq -factual questions (fq). in fq, answer entities are connected to question entities via short paths (up to 3 steps) in the wikidata kb @xcite . using fq, we analyze a simple but effective approach to incorporating kb knowledge into a textual qa system. we convert kb paths to text (using surface forms of entities and relation) and append it to the textual passage as additional context for a bert-based @xcite qa system. we first establish an upper bound oracle setting by building a retriever that provides the shortest path to an answer. we show that, in the presence of such knowledge, our approach leads to significant gains (up to 6 f1 for short-answers, 8-9 f1 for multi-hop questions). we experiment with several variants of kb path-retrieval methods and show that retrieving good paths is difficult: previouslyused personalized pagerank (haveliwala, 2003) (ppr)-based methods find answer entities less than 30% of the time, and even our weakly-supervised improvements recall answer entities no more than 40% of the time. as a consequence injecting retrieved kb paths in a realistic qa setting like nq yields only small, inconsistent improvements. to summarize our contributions, we (1) identify a new experimental subset of nq that supports (2) the study of effectiveness of kb path-retrieval approaches. we also (3) describe a simple, modelagnostic method to using oracle kb paths that can significantly improve qa performance and evaluate ppr based path-retrieval methods. to our knowledge this is the first study of such approaches on a qa dataset not curated for kbqa. ## dataset the natural questions (nq) dataset @xcite is a large scale qa dataset containing 307,373 training, 7,830 dev, and 7,842 test examples. each example is a user query paired with wikipedia documents annotated with a passage (long answer) answering the question and one or more short spans (short answer) containing the answer. the questions in nq are not artificially constructed, making the nq task more difficult @xcite . we use sling @xcite (which uses an np chunker and phrase table for linking entities to wikidata) to entity link the questions and documents. to focus on knowledge-driven factoid question answering, we create a subset of nq having relevant knowledge in the kb. shortest paths between entities in kb is very often used as a proxy for gold knowledge linking questions to answer @xcite and we use the same proxy in our setting. specifically, we select questions whose short answers are entities in the kb and have a short path (up to 3 steps) from a question entity to an answer entity. these paths contain knowledge relevant to the question but are not necessarily the right path to answer the question. we call this subset factual questions (fq) containing 6977 training, 775 dev and 264 (83 1-hop, 97 2-hop and 84 3-hop) test samples. fq being an entity centric subset of nq, provides a setting to investigate augmenting kb paths for real-world factoid question for which relevant knowledge exists in the kb. examples of the dataset are provided in ## model given a question q, our knowledge retriever extracts top facts from a kb. we represent them in natural language form and augment it to a standard bert model for reading comprehension as additional context along with the passage p . ## experiments and results setup: as every passage that doesn't contain the answer is a potential negative, we sample a subset of negatives to balance the dataset. for the factual nq subset, we sample 2% of the negatives as in @xcite to enable faster training. we find that increasing the negatives to 10% improves results by ∼2 points and hence for a fair comparison, we sample 10% of the negatives for our models and the reimplemented baseline on the full nq dataset. we use the same preprocessing steps and all other hyperparameter settings as in @xcite . ## conclusion we investigate incorporating kb facts into a realworld qa -natural questions. we create a subset of nq, factual questions, to facilitate evaluation of kb integration. we present an oracle setting, where the gold kb path is provided and establish a strong upper-bound. we experimentally show that ppr based retrievers have low recall of answer entities and do not improve downstream qa showing that path-retrieval is a bottleneck for kb integration.