End-to-End Beam Retrieval for Multi-Hop Question Answering (2024)

Abstract: Multi-hop question answering (QA) involves finding multiple relevant passages and step-by-step reasoning to answer complex questions, indicating a retrieve-and-read paradigm. However, previous retrievers were customized for two-hop questions, and most of them were trained separately across different hops, resulting in a lack of supervision over the entire multi-hop retrieval process and leading to poor performance in complicated scenarios beyond two hops. In this work, we introduce Beam Retrieval, an end-to-end beam retrieval framework for multi-hop QA. This approach models the multi-hop retrieval process in an end-to-end manner by jointly optimizing an encoder and two classification heads across all hops. Moreover, Beam Retrieval maintains multiple partial hypotheses of relevant passages at each step, expanding the search space and reducing the risk of missing relevant passages. To establish a complete QA system, we incorporate a supervised reader or a large language model (LLM). Experimental results demonstrate that Beam Retrieval achieves a nearly 50% improvement compared with baselines on challenging MuSiQue-Ans, and it also surpasses all previous retrievers on HotpotQA and achieves 99.9% precision on 2WikiMultiHopQA. Providing high-quality context, Beam Retrieval helps our supervised reader achieve new state-of-the-art performance and substantially improves the few-shot QA performance of LLMs.

## introduction question answering (qa) has been a mainstream research in natural language processing (nlp) for a long time. with the development of pretrained language models (plms), simple qa tasks can be solved by adopting a bert-like plm @xcite , 2019). as a result, researchers have been increasingly drawn to more complex qa benchmarks, such as multi-hop qa. this presents a significant challenge, as it requires reasoning across multiple and diverse passages to accurately answer complicated multi-hop questions. many high-quality multi-hop qa datasets have been introduced, such as hotpotqa @xcite , 2wikimulti-hopqa @xcite , musique @xcite and so on. figure 1 illustrates an example of an actual question taken from musique-ans dataset. mainstream methods for multi-hop qa often follow a retrieve-and-read paradigm @xcite , including a passage retriever to filter out extraneous information and a reader to obtain the final answer @xcite @xcite @xcite @xcite . however, these methods have primarily focused on two-hop scenarios, exhibiting limited adaptability to more complex situations beyond two-hops. additionally, while multi-hop retrieval requires identifying the next hop passage based on the question and previously selected passages (see figure ?? ), few of them focus on supervision over the entire retrieval process. furthermore, these retrievers exhibit limited robustness, as the entire retrieval process is susceptible to failure if the first stage identifies irrelevant passages. in conclusion, previous retrievers perform poorly when handling questions with more than 2 hops and provide low-quality context for downstream qa tasks. to address the described problems, we propose beam retrieval, an end-to-end beam retrieval framework for multi-hop qa. beam retrieval utilizes an encoder and two classification heads to model the entire multi-hop retrieval process in an end-to-end manner and can be adapted to a question with a variable hop. during training, beam retrieval accumulates the loss at each step and jointly optimizes the encoder and two classification heads in the backpropagation phase, enabling the model to learn the entire retrieval process. during inference, beam retrieval searches the relevant passage at each step until the highest predicted score falls below a predefined threshold. in summary, beam retrieval produces a chain of relevant passages with the highest score using a single forward pass, effectively learning the entire multi-hop retrieval process. moreover, we employ the beam search paradigm by keeping track of multiple partial hypotheses of relevant passages at each step. this approach enables our model to learn more negative passage pairs in the expanded search space, enhances the probability of obtaining the truly relevant passages, and mitigates the impact of retrieval errors that may occur in the early stages. to reduce the gap between training and reasoning, beam retrieval is designed to reason using the same beam size as it employs during training. beam retrieval can also serve as a plugin in the qa domain, providing high-quality relevant context and enhancing the performance of downstream qa tasks. based on beam retrieval, we implement a multi-hop qa system to extract the answers by incorporating a supervised reader @xcite following conventional machine reading comprehension setting or a few-shot large language model (llm) @xcite . we validate beam retrieval by extensive experiments on three benchmark datasets musique-ans, hotpotqa and 2wikimultihopqa, and experimental results demonstrate that beam retrieval surpasses all previous retrievers by a large margin. consequently, beam retrieval substantially improves the qa performance of downstream qa readers on all three datasets. we highlight our contributions as follows: notice that the unselected passages in the first stage will not be utilized in the second stage, leaving limitations in retrieval. the beam retrieval proposed in this paper, primarily aimed at the reading comprehension setting, similarly introduces the idea of beam search as in beamdr. however, unlike beamdr, beam retrieval emphasizes modeling the entire multi-hop retrieval process and dealing with complex scenarios beyond two hops. ## beam retrieval beam retrieval is designed to handle a k-hop multihop question q and accurately selects the most relevant passages, providing nearly noiseless context for downstream qa tasks. in this section, we clarify how beam retrieval infers and trains in an end-to-end manner, which is illustrated in figure 2 . ## conclusion we present beam retrieval, an end-to-end beam retrieval framework for multi-hop qa. this approach models the entire retrieval process in an end-to-end manner and maintains multiple partial hypotheses of relevant passages at each step, showing great performance in complex scenarios beyond two hops. experimental results on three datasets prove the effectiveness of beam retrieval and demonstrate it could substantially improve the qa performance of downstream readers. in general, beam retrieval establishes a strong baseline for complex multi-hop qa, where we hope that future work could explore more advanced solutions.