Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval (2021)

Abstract: Complex question answering often requires finding a reasoning chain that consists of multiple evidence pieces. Current approaches incorporate the strengths of structured knowledge and unstructured text, assuming text corpora is semi-structured. Building on dense retrieval methods, we propose a new multi-step retrieval approach (BeamDR) that iteratively forms an evidence chain through beam search in dense representations. When evaluated on multi-hop question answering, BeamDR is competitive to state-of-the-art systems, without using any semi-structured information. Through query composition in dense space, BeamDR captures the implicit relationships between evidence in the reasoning chain. The code is available at https://github.com/henryzhao5852/BeamDR .

## introduction answering complex questions requires combining knowledge pieces through multiple steps into an evidence chain (ralph @xmath0. when the available knowledge sources are graphs or databases, constructing chains can use the sources' inherent structure. however, when the information needs to be pulled from unstructured text (which often has better coverage), standard information retrieval (ir) approaches only go "one hop": from a query to a single passage. recent approaches @xcite @xcite try to achieve the best of both worlds: use the unstructured text of wikipedia with its structured hyperlinks. while they show promise on benchmarks, it's difficult to extend them beyond academic testbeds because real-world datasets often lack this structure. for example, medical records lack links between reports. dense retrieval @xcite @xcite promising path to overcome this limitation. it encodes the query and evidence (passage) into dense vectors and matches them in the embedding space. in addition to its efficiency-thanks to maximum inner-product search (mips) @xcite show that dense retrieval rivals bert @xcite -based (sparse) retrieve-then-rerank ir pipelines on single step retrieval. unlike traditional term-based retrieval, fully learnable dense encodings provide flexibility for different tasks. this paper investigates a natural question: can we build a retrieval system to find an evidence chain on unstructured text corpora? we propose a new multi-step dense retrieval method to model the implicit relationships between evidence pieces. we use beam search (section 2) in the dense space to find and cache the most relevant candidate chains and iteratively compose the query by appending the retrieval history. we improve the retrieval by encouraging the representation to discriminate hard negative evidence chains from the correct chains, which are refreshed by the model. we evaluate beam dense retrieval (beamdr) on hotpotqa @xcite , a multi-hop question answering benchmark. when retrieving evidence chains directly from the corpus (full retrieval), beamdr is competitive to the state-of-the-art cascade reranking systems that use wikipedia links. combined with standard reranking and answer span extraction modules, the gain from full retrieval propagates to findings answers (section 3). by iteratively composing the query representation, beamdr captures the hidden "semantic" relationships in the evidence (section 4). ## beamdr: beam dense retriever this section first discusses preliminaries for dense retrieval, then introduces our method, beamdr. ## experiments: retrieval and answering our experiments are on hotpotqa fullwiki setting @xcite , the multi-hop question answering benchmark. we mainly evaluate on retrieval that extracts evidence chains (passages) from the corpus; we further add a downstream evaluation on whether it finds the right answer. ## exploring how we hop in this section, we explore how beamdr constructs evidence chains. ## related work extracting multiple pieces of evidence automatically has applications from solving crossword puzzles @xcite , graph database construction @xcite , and understanding relationships @xcite to question answering @xcite , which is the focus of this work. given a complex question, researchers have investigated multi-step retrieval techniques to find an evidence chain. knowledge graph question answering approaches @xcite directly search the evidence chain from the knowledge graph, but falter when kg coverage is sparse. with the release of large-scale datasets @xcite , recent systems @xcite @xcite , inter alia) use wikipedia abstracts (the first paragraph of a wikipedia page) as the corpus to retrieve the evidence chain. @xcite treat wikipedia as a knowledge graph, where each entity is identified by its textual span mentions, while other approaches @xcite directly retrieve passages. they first adopt a single-step retrieval to select the first hop passages (or entity mentions), then find the next hop candidates directly from wikipedia links and rerank them. like beamdr, @xcite use beam search to find the chains but still rely on a graph neural network over wikipedia links. beamdr retrieves evidence chains through dense representations without relying on the corpus semi-structure. @xcite iteratively generate the query from the question and retrieved history, and use traditional sparse ir systems to select the passage, which complements beamdr's approach. ## conclusion we introduce a simple yet effective multi-step dense retrieval method, beamdr. by conducting beam search and globally refreshing negative chains during training, beamdr finds reasoning chains in dense space. beamdr is competitive to more complex sota systems albeit not using semi-structured information. while beamdr can uncover relationship embedded within a single question, future work should investigate how to use these connections to resolve ambiguity in the question @xcite , resolve entity mentions @xcite , connect concepts across modalities @xcite , or to connect related questions to each other @xcite .