Combining Lexical and Dense Retrieval for Computationally Efficient Multi-hop Question Answering (2021)

Abstract: In simple open-domain question answering (QA), dense retrieval has become one of the standard approaches for retrieving the relevant passages to infer an answer. Recently, dense retrieval also achieved state-of-the-art results in multi-hop QA, where aggregating information from multiple pieces of information and reasoning over them is required. Despite their success, dense retrieval methods are computationally intensive, requiring multiple GPUs to train. In this work, we introduce a hybrid (lexical and dense) retrieval approach that is highly competitive with the state-of-the-art dense retrieval models, while requiring substantially less computational resources. Additionally, we provide an in-depth evaluation of dense retrieval methods on limited computational resource settings, something that is missing from the current literature.

## introduction multi-hop qa requires retrieval and reasoning over multiple pieces of information @xcite . for instance, consider the multi-hop question: "where is the multinational company founded by robert smith headquartered?". to answer this question we first need to retrieve the passage about robert smith in order to find the name of the company he founded (general mills), and subsequently retrieve the passage about general mills, which contains the answer the question @xcite . even though multi-hop qa requires multiple retrieval hops, it is fundamentally different from session search @xcite and conversational search @xcite @xcite , since in multi-hop qa the information need of the user is expressed in a single question, thus not requiring multiple turns of interaction. qa systems typically consist of (i) a retriever that identifies the passage/document in the underlying collection that contains the answer to the user's question, and (ii) a reader that extracts or generates the answer from the identified passage @xcite . given that often the answer cannot be found in the top-ranked passage, inference follows a standard beam-search procedure, where top-k passages are retrieved and the reader scores are computed for all k passages @xcite . however, readers are very sensitive to noise in the top-k passages, thus making the performance of the retriever critical for the performance of qa systems @xcite . this is further amplified in multi-hop qa, where multiple retrieval hops are performed; potential retrieval errors get propagated across hops and thus severely harm qa performance. the majority of current approaches to multihop qa use either traditional ir methods (tf-idf, bm25) @xcite or graph-based methods for the retriever @xcite . however, those approaches have serious limitations. the former approaches require high lexical overlap between questions and relevant passages, while the latter rely on an interlinked underlying corpus, which is not always the case. recently, @xcite introduced a dense multi-hop passage retrieval model that constructs a new query representation based on the question and previously retrieved passages and subsequently uses the new representation to retrieve the next set of relevant passages. this model achieved state-of-the-art results while not relying on an interlinked underlying corpus. even though dense retrieval models achieve state-of-the-art results on multi-hop qa, they are computationally intensive, requiring multiple gpus to train. existing work only reports results for the cases where such resources are available; therefore providing no answer on how feasible is to train such models on a low resource setting. in this paper, we focus on developing an efficient retriever for multi-hop qa that can be trained effectively in a low computational resource setting. we aim to answer the following research questions: rq1 how does the performance of dense retrieval compare to lexical and hybrid approaches? rq2 how does the performance degrade in the low computational resource settings? our main contributions are the following: (i) we propose a hybrid (lexical and dense) retrieval model which is competitive against its fully dense competitors while requiring eight times less computational power and (ii) we perform a thorough analysis on the performance of dense passage retrieval models on the task of multi-hop qa. foot_0 2 task description let @xmath0, and q a multi-hop question. given q and c the task is to retrieve a set of relevant passages @xmath1, p 2 , . . . , p n }, where p @xmath2. in the multi-hop scenario we consider here, not all relevant passages can be retrieved using the input question q alone. this is due to the fact that there is a low lexical overlap or semantic relationship between question q and one or more of the relevant passages in p. in this case, information from one of the relevant passages p i is needed to retrieve another relevant passage p j , where p j may be lexically/semantically different from question q. ## experimental setup in this section, we describe the dataset used in our experiments, the metrics we use to answer our research questions and the models we compare against. ## results in this section, we present our experimental results that answer our research questions. in the first group we show single-hop models, while in the second group we show multi-hop models. ## conclusion in this work, we provided insights on the performance of state-of-the-art dense retrieval for multihop questions. we showed that rerank+dpr 2 (our hybrid model) outperforms mdr (the state-of-theart multi-hop dense retrieval model) in the low re-source setting, and it is competitive with mdr in the setting where mdr uses considerably more computational resources. finally, we highlighted that fully dense retrieval models get harmed when using limited computational resources. for future work, we plan to build on our insights to improve the performance of multi-hop models by combining the strengths of lexical and dense retrieval. also, we aim to develop less computationally expensive multi-hop retrieval models.