Answering Open-Domain Questions of Varying Reasoning Steps from Text (2021)

Abstract: We develop a unified system to answer directly from text open-domain questions that may require a varying number of retrieval steps. We employ a single multi-task transformer model to perform all the necessary subtasks—retrieving supporting facts, reranking them, and predicting the answer from all retrieved documents—in an iterative fashion. We avoid crucial assumptions of previous work that do not transfer well to real-world settings, including exploiting knowledge of the fixed number of retrieval steps required to answer each question or using structured metadata like knowledge bases or web links that have limited availability. Instead, we design a system that can answer open-domain questions on any text collection without prior knowledge of reasoning complexity. To emulate this setting, we construct a new benchmark, called BeerQA, by combining existing one- and two-step datasets with a new collection of 530 questions that require three Wikipedia pages to answer, unifying Wikipedia corpora versions in the process. We show that our model demonstrates competitive performance on both existing benchmarks and this new benchmark. We make the new benchmark available at https://beerqa.github.io/ .

## introduction using knowledge to solve problems is a hallmark of intelligence. since human knowledge is often containned in large text collections, open-domain question answering (qa) is an important means for intelligent systems to make use of the knowledge in large text collections. with the help of large-scale datasets based on wikipedia @xcite and other large corpora @xcite @xcite , the research community has made substantial progress on tackling this problem in recent years, including * these authors contributed equally. in the direction of complex reasoning over multiple pieces of evidence, or multi-hop reasoning @xcite @xcite . despite this success, most previous systems are developed with, and evaluated on, datasets that contain exclusively single-hop questions (ones that require a single document or paragraph to answer) or two-hop ones. as a result, their design is often tailored exclusively to single-hop (e.g., @xcite or multi-hop questions (e.g., @xcite @xcite @xcite . even when the model is designed to work with both, it is often trained and evaluated on exclusively single-hop or multi-hop settings (e.g., @xcite . in practice, not only can we not expect open-domain qa systems to receive exclusively single-or multi-hop questions from users, but it is also non-trivial to judge reliably whether a question requires one or multiple pieces of evidence to answer a priori. for instance, "in which u.s. state was facebook founded?" appears to be single-hop, but its answer cannot be found in the main text of a single english wikipedia page. besides the impractical assumption about reasoning hops, previous work often also assumes access to non-textual metadata such as knowledge bases, entity linking, and wikipedia hyperlinks when retrieving supporting facts, especially in answering complex questions @xcite @xcite @xcite . while this information is helpful, it is not always available in text collections we might be interested in getting answers from, such as news or academic research articles, besides being labor-intensive and time-consuming to collect and maintain. it is therefore desirable to design a system that is capable of extracting knowledge from text without using such metadata, to maximally emphasize using knowledge available to us in the form of text. ## open-domain question answering the task of open-domain question answering is concerned with finding the answer to a question from a large text collection d. successful solutions to this task usually involve two crucial components: an information retrieval system that finds a small set of relevant documents d from d, and a reading comprehension system that extracts the answer from it. 3 @xcite presented the first neuralnetwork-based approach to this problem, which was later extended by @xcite with a reranking system to further reduce the amount of context the reading comprehension component has to consider to improve answer accuracy. more recently, @xcite showed that this single-step retrieve-and-read approach to opendomain question answering is inadequate for more complex questions that require multiple pieces of evidence to answer (e.g., "what is the population of mark twain's hometown?"). while later work approaches these by extending supporting fact retrieval beyond one step, most assumes that all questions are either exclusively single-hop or multihop during training and evaluation. we propose irrr, a system that performs variable-hop retrieval for open-domain qa to address these issues, and present a new benchmark, beerqa, to evaluate systems in a more realistic setting. ## irrr: iterative retriever, reader, and reranker in this section, we present a unified model to perform all of the subtasks necessary for open-domain question answering-iterative retriever, reader, and reranker (irrr), which performs the subtasks involved in an iterative manner to accommodate questions with a varying number of steps. irrr aims at building a reasoning path from the question , through all the necessary supporting documents or @xmath0. 4 as shown in figure 1 , irrr operates in a loop of retrieval, reading, and reranking to expand the reasoning path with new documents @xmath1. specifically, given a question , we initialize the reasoning path with the question itself, i.e., 0 = [ ], and generate from it a search query with irrr's retriever. once a set of relevant documents d 1 ⊂ d is retrieved, they might either help answer the question, or reveal clues about the next piece of evidence we need to answer . the reader model then attempts to read each of the documents in d 1 to answer the question combined with the current reasoning path . if more than one answer can be found from these candidate reasoning paths, we predict the answer with the highest answerability score, which we will detail in section 3.2. if no answer can be found, then irrr's reranker scores each retrieved paragraph against the current reasoning path, and appends the top-ranked paragraph to the current reasoning path, 4 for simplicity, we assume that there is a single set of relevant supporting facts that helps answer each question. retriever (query generator) reader reranker 4-way clsf [cls] q [sep] title0 [cont] ctx0 [sep] … transformer-encoder h[cls] h1 h2 h3 h4 h5 h6 … token-wise binary prediction span prediction rerank rerank rerank nce clsf …… ✓ / ✗ noanswer / span / yes / no ✓ / ✗ ✓ / ✗ start end gold paragraph figure 2: the overall architecture of our irrr model, which uses a shared transformer encoder to perform all subtasks of open-domain question answering. to reduce computational cost and improve model representations of reasoning paths from shared statistical learning, irrr is implemented as a multitask model built on a pretrained transformer model that performs all three subtasks. at a high level, it consists of a transformer encoder @xcite which takes the reasoning path (the question and all retrieved paragraphs so far) as input, and one set of task-specific parameters for each task of retrieval, reranking, and reading comprehension (see figure 2 ). the retriever generates natural language search queries by selecting words from the reasoning path, the reader extracts answers from the reasoning path and abstains if its confidence is not high enough, and the reranker assigns a scalar score for each retrieved paragraph as a potential continuation of the current reasoning path. the input to our transformer encoder is formatted similarly to that of the bert model @xcite . for a reasoning path that consists of the question and retrieved paragraphs, the input is formatted as " @xcite question [sep] title 1 [cont] para 1 [sep] . . . title [cont] para [sep]", where [cls], [sep], and [cont] are special tokens to separate different components of the input. the [cont] embedding is randomly initialized with a truncated normal distribution with a standard deviation of 0.02, and finetuned with other model parameters during training. we will detail each of the task-specific components in the following subsections. ## results in this section, we present the performance of irrr when evaluated against previous systems on standard benchmarks, and demonstrate its efficacy on our new, unified benchmark, especially with the help of iterative training. ## related work the availability of large-scale question answering (qa) datasets has greatly contributed to the research progress on open-domain qa. squad @xcite the ingerophrynus gollum is named after a character in a book that sold how many copies? ingerophrynus is a genus of true toads with 12 species. ... in 2007 a new species, "ingerophrynus gollum", was added to this genus. this species is named after the character gollum created by j. r. r. tolkien." query ingerophrynus gollum book sold copies j. r. r. tolkien ## conclusion in this paper, we presented iterative retriever, reader, and reranker (irrr), a system that uses a single model to perform subtasks to answer open-domain questions of arbitrary reasoning steps. irrr achieves competitive results on standard opendomain qa benchmarks, and establishes a strong baseline on beerqa, the new unified benchmark we present, which features questions with mixed levels of complexity.