Grape: Knowledge Graph Enhanced Passage Reader for Open-domain Question Answering (2022)

Abstract: A common thread of open-domain question answering (QA) models employs a retriever-reader pipeline that first retrieves a handful of relevant passages from Wikipedia and then peruses the passages to produce an answer. However, even state-of-the-art readers fail to capture the complex relationships between entities appearing in questions and retrieved passages, leading to answers that contradict the facts. In light of this, we propose a novel knowledge graph enhanced passage reader, namely Grape, to improve the reader performance for open-domain QA. Specifically, for each pair of question and retrieved passage, we first construct a localized bipartite graph, attributed to entity embeddings extracted from the intermediate layer of the reader model. Then, a graph neural network learns relational knowledge while fusing graph and contextual representations into the hidden states of the reader model. Experiments on three open-domain QA benchmarks show Grape can improve the state-of-the-art performance by up to 2.2 exact match score with a negligible overhead increase, with the same retriever and retrieved passages. Our code is publicly available at https://github.com/jumxglhf/GRAPE.

## introduction open-domain question answering (qa) tasks aim to answer questions in natural language based on large-scale unstructured passages such as wikipedia @xcite . a common thread of modern open-domain qa models employs a retriever-reader pipeline, in which a retriever aims to retrieve a handful of relevant passages w.r.t. a given question, and a reader aims to infer a final answer from the received passages @xcite @xcite . although these methods have achieved remarkable advances on various open-domain qa benchmarks, the state-of-the-art readers, such as fid @xcite , still often produce answers that contradict the facts. as shown in figure 1 , the fid reader fails to produce correct answers due to inaccurate understanding of the factual evidence. therefore, instead of improving the retrievers to saturate the readers with higher answer coverage in the retrieved passages @xcite @xcite , in this work, we aim at improving the readers by leveraging structured factual triples from the knowledge graph (kg). a knowledge graph, such as wikidata (vrandečić and krötzsch, 2014), contains rich relational information between entities, many of which can be further mapped to corresponding mentions in questions and retrieved passages. to verify the possible improvements brought by the kg, we conduct a simple analysis to examine the percentage of related fact triples present on the kg in the data, i.e., entities in questions are neighbors of answer entities in retrieved passages through any relation. we also wonder how many of the above examples are correctly answered by state-of-the-art readers. in this paper, we propose a novel knowledge graph enhanced passage reader, namely grape, to improve the reader performance for opendomain qa. considering the enormous size of kgs and complex interweaving between entities (e.g., over 5 million entities and over 30 neighbors per entity on wikidata), direct reasoning on the entire graph is intractable. thus, we first construct a localized bipartite graph for each pair of question and passage, where nodes represent entities contained within them, and edges represent relationships between entities. then, node representations are initialized with the hidden states of the corresponding entities, extracted from the intermediate layer of the reader model. next, a graph neural network learns node representations with relational knowledge, and passes them back into the hidden states of the reader model. through this carefully curated design, grape takes into account both aspects of knowledge as a holistic framework. to the best of our knowledge, we are the first work to leverage knowledge graphs to enhance the passage reader for open-domain qa. our experiments demonstrate that, given the same retriever and the same set of retrieved passages, grape can achieve superior performance on three opendomain qa benchmarks (i.e., nq, triviaqa, and webq) with up to 2.2 improvement on the exact match score over the state-of-the-art readers. in particular, our proposed grape nearly doubles the improvement gain on the subset that can be enhanced by fact triplets on the kg. ## related work text-based open-domain qa mainstream opendomain qa models employ a retriever-reader architecture, and recent follow-up work has mainly focused on improving the retriever or the reader @xcite . for the retriever, most of them split text paragraphs on wikipedia pages into over 20 million disjoint chunks of 100 words, each of which is called a passage. traditional methods such as tf-idf and bm25 explore sparse retrieval strategies by matching the overlapping contents between questions and passages @xcite . dpr @xcite revolutionized the field by utilizing dense contextualized vectors for passage indexing. furthermore, other research improved the performance by better training strategies @xcite , passage re-ranking @xcite or directly generating passages @xcite . whereas for the reader, extractive readers aimed to locate a span of words in the retrieved passages as answer @xcite @xcite . on the other hand, fid and rag, current state-of-the-art readers, leveraged encoder-decoder models such as t5 to generate answers @xcite . nevertheless, these readers only used text corpus, failing to capture the complex relationships between entities, and hence resulting in produced answers contradicting the facts. ## proposed method: grape in this section, we elaborate on the details of the proposed grape. figure 3 shows its overall architecture. grape adopts a retriever-reader pipeline. specifically, given a question, it first utilizes dpr to retrieve top-k relevant passages from wikipedia ( §3.1). then, to peruse the retrieved passages, it constructs a localized bipartite graph for each pair of question and passage ( §3.2.1). the constructed graphs possess tractable yet rich knowledge about the facts among connected entities. finally, with the curated graphs, structured facts are learned through a relation-aware graph neural network (gnn) and fused into token-level representations of entities in the passages ( §3.2.2). ## experiments in this section, we conduct comprehensive experiments on three community-acknowledged public open-domain qa benchmarks: natural questions (nq) based on google search queries, triviaqa based on questions from trivia and quiz-league websites, and web questions (webq) based on questions from google suggest api @xcite @xcite . we explore the same train / dev / test splits and preprocessing techniques as used by @xcite . ## conclusion in this work, we study the problem of open-domain qa. we discover that state-of-the-art readers fail to capture the complex relationships between entities appearing in questions and retrieved passages, resulting in produced answers that contradict the facts.