Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge (2022)

Abstract: We propose a novel open-domain question answering (ODQA) framework for answering single/multi-hop questions across heterogeneous knowledge sources.The key novelty of our method is the introduction of the intermediary modules into the current retriever-reader pipeline.Unlike previous methods that solely rely on the retriever for gathering all evidence in isolation,our intermediary performs a chain of reasoning over the retrieved set.Specifically, our method links the retrieved evidence with its related global context into graphs and organizes them into a candidate list of evidence chains.Built upon pretrained language models, our system achieves competitive performance on two ODQA datasets, OTT-QA and NQ, against tables and passages from Wikipedia.In particular, our model substantially outperforms the previous state-of-the-art on OTT-QA with an exact match score of 47.3 (45% relative gain).

## introduction the task of open-domain question answering (odqa) typically involves multi-hop reasoning, such as finding relevant evidence from knowledge sources, piecing related evidence with context together, and then producing answers based on the final supportive set. while many questions can be answered by a single piece of evidence @xcite , answering more complex questions are of great interest and require reasoning beyond local document context @xcite . the problem becomes more challenging when evidence is scattered across heterogeneous sources, e.g., unstructured text and structured tables @xcite , which necessitates hopping from one knowledge modality to another. consider the question in fig. 2 . to form the final answer, a model needs to find the entry list (a table) of the mentioned touring car race, look up the driver name with the correct rank, search for the corresponding driver information, and extract the birthplace from the free-form text. existing retriever-reader methods @xcite for odqa mainly customize the retriever model for tackling individual question types, i.e., exclusively relying on the retriever to gather all necessary context in a query-dependent fashion. as shown in fig. 1 , the single-hop model @xcite ) only retrieves a list of isolated passages (blue dots). for multi-hop cases, an iterative retriever looks for a query-dependent path of passages (blue-purple dot chains) @xcite , i.e., the later hop passages are retrieved using expanded queries including the original question and previous hop passages. although those retrieval-only methods achieve promising results on their targeted cases, the customized retrievers are unable to generalize well. for example, an iterative passage retriever trained with both multi-hop and single-hop questions performs poorly over both types @xcite . for real-world applica-figure 2 : an illustration of core for odqa. given a question, the retriever first finds hop-1 evidence from the entire wikipedia (orange arrows). then the linker gathers relevant documents for the hop-1 evidence (purple arrows), which are treated as hop-2 evidence. next, the chainer reranks all hop-1 and hop-2 evidence and splices them together into evidence chains (blue arrows). finally, the reader takes in the top-50 chains and produces the answer (black arrows). the gold evidence chain is marked in red. we propose a novel chain of reasoning (core) framework that generalizes well on both singlehop and multi-hop question answering. the main contribution of core is the introduction of two intermediary modules, the linker and the chainer, that play the bridging role between the retriever and the reader, i.e., piecing together related evidence with necessary context for single/multi-hop questions. these two modules work in a forward-backward fashion. in the forward pass, the linker, a novel table entity linking model ( §3.1), links the raw evidence with its related context (e.g., table-passage graphs in fig. 1 ). the chainer, a new unsupervised reranker ( §3.2), then prunes all linked evidence using the corresponding question generation scores from a pretrained language model @xcite to form a shortlist of relevant evidence chains in a backward noisy channel fashion (e.g., table-passage paths in fig. 1 ). by delegating the hopping operation to the intermediary, our formalization can potentially gather evidence more effectively over different question types. to demonstrate the effectiveness of core, we evaluated the proposed model on two popular odqa datasets, the multi-hop dataset ott-qa @xcite and the single-hop dataset nq @xcite . empirically, we show that our approach is general for both types of reasoning in odqa. in particular, core substantially outperforms the previous state-of-the-art (sota) on ott-qa by 14+ points on exact match scores (45%+ relative gain), and it is competitive with sota models on nq. moreover, we show that one single unified model can learn to solve both tasks under our framework. from our analysis, we also find that our evidence chains can potentially help answer single-hop questions by enriching the evidence with more supportive context. foot_0 ## overview of the core framework the core framework is designed to answer questions where the answer is a contiguous span from a table t or a passage p. here neither t nor p is given, so they need to be retrieved from the table corpus c t and the passage corpus c p , respectively. for single-hop questions, a single t or p may be sufficient, whereas for multi-hop questions, one or more t and p are required to find the answer. as shown in fig. 2 , core consists of a retriever, a linker, a chainer and a reader. we adopt the dpr model @xcite as our retriever. we only briefly describe the retriever here as it is not the main focus of our work. the dpr is a bi-encoder model that consists of a question encoder and a context encoder. for our setting, the questions and passages/tables are represented by the [cls] embedding produced by their respective encoder, and the retrieval is done by maximum inner product search in the vector space. for a given question, we use dpr to retrieve the initial evidence set which includes tables and passages. given the initial evidence set (e.g., the car race entry list table in fig. 2 ), our intermediary module produces a list of query-dependent evidence chains (e.g., the red line linked evidence chain consisting of the car race entry list and the driver's wikipedia page). we first propose a linker model ( §3.1) to expand the candidate evidence set by including extra passages related to tables in the initial set (purple arrows in fig. 2 ). this step allows the model to enrich the evidence context, especially including reasoning chains needed for multi-hop questions. since there could be many links between a piece of evidence and others (i.e., a densely connected graph), considering all links is computationally infeasible for the downstream reader. thus, we develop a chainer model ( §3.2) to prune the evidence graph with the corresponding question and then chain the evidence across hops together to form query-dependent paths. here, we only keep top-k scored chains for reading so that the reader can work on a fixed computation budget. finally, the fusion-in-decoder (fid) @xcite , a t5-based generative model @xcite , is used as the reader for generating the final answer. the model first encodes each top-k evidence chain independently along with the question. during decoding, the decoder can attend to all chains, thus fusing all the input information. ## intermediary modules in this part, we present the two key components of core for supporting multi-hop reasoning, i.e., the linker for building evidence graphs and the chainer for forming query-dependent paths. ## experiments in this section, we first describe datasets and knowledge sources ( §4.1). then we discuss single-set and joint-set experiment setups ( §4.2) and baselines for comparison ( §4.3). finally, we present experiment results on ott-qa and nq ( §4.4). ## analysis in this section, we conduct ablation studies and analyze different components of our framework. ## related work multi-hop question answering (qa) has been studied extensively for both knowledge base (kb) setting @xcite @xcite and open-domain setting @xcite @xcite . for kb-based setting, the models are trained to parse questions into logical forms that can be executed against kb @xcite or directly select entities in kb as answers @xcite . on the other hand, the open-domain setting requires the model to retrieve q&a evidence chain q: who plays ryders mum table title: list of home and away characters on home and away || character, actor(s), duration || ryder jackson, lukas radovich, 2017-|| ryder jackson is a: lara cox a fictional character from the australian television soap opera "home and away" ... his mother is quinn jackson (lara cox), who is estranged from alf. when ... q: what was blur 's first table title: list of uk top-ten singles in 1994 number one single in the uk || artist, weeks, singles || blur, 3, "girls & boys", "parklife" || girls & boys (blur song) is a 1994 a: country house song by british rock band blur. it was released ... "girls & boys" was blur's first top 5 hit and their most successful single until "country house" reached number 1 the following year ... q: who is the dad of the cyclist table title: holland hills classic that placed directly behind || year, first, second, third || 2011, marianne vos, marieke van wanroij, jessie daams || marieke van wanroij at the jessie daams jessie daams ( born 28 may 1990 ) is a belgian racing cyclist . 2011 holland hills classic ? she competed in the 2013 uci women 's road race in florence . a: hans daams her father is the dutch cyclist hans daams . q: what other team did the table title: 2012 charlotte eagles season cuban player on the 2012 char-|| no, position, player, nation || 19, midfielder, miguel ferrer, cuba || miguel ferrer lotte eagles team play for ? (footballer) miguel ferrer ( born march 28 , 1987 ) is a cuban footballer who played for a: wichita wings the wichita wings in the major indoor soccer league . ## conclusion in this paper, we present a new framework, core, for odqa over heterogeneous knowledge sources. with the novel task-agnostic intermediary module, core can effectively handle single/multi-hop tasks using a unified model and achieve new sota results on ott-qa. our analyses show that the intermediary module is necessary to achieve good results. for future work, it would be interesting to apply our framework to other complex reasoning tasks such as fact verification @xcite or commonsense qa @xcite .