Hybrid Hierarchical Retrieval for Open-Domain Question Answering (2023)

Abstract: Retrieval accuracy is crucial to the performance of open-domain question answering (ODQA) systems. Recent work has demonstrated that dense hierarchical retrieval (DHR), which retrieves document candidates first and then relevant passages from the refined document set, can significantly outperform the single stage dense passage retriever (DPR). While effective, this approach requires document structure information to learn document representation and is hard to adopt to other domains without this information. Additionally, the dense retrievers tend to generalize poorly on out-of-domain data comparing with sparse retrievers such as BM25. In this paper, we propose Hybrid Hierarchical Retrieval (HHR) to address the existing limitations. Instead of relying solely on dense retrievers, we can apply sparse retriever, dense retriever, and a combination of them in both stages of document and passage retrieval. We perform extensive experiments on ODQA benchmarks and observe that our framework not only brings in-domain gains, but also generalizes better to zero-shot TriviaQA and Web Questions datasets with an average of 4.69% improvement on recall@100 over DHR. We also offer practical insights to trade off between retrieval accuracy, latency, and storage cost. The code is available on github.

## introduction open-domain question answering (odqa) @xcite ) aims to answer questions based on a large corpus without pre-specified context, and enjoys a broad scope of real-world applications such as chatbots, virtual assistants, search engines, etc. recent odqa systems often follow a two stage retrieve-then-read architecture @xcite @xcite . given a question, a retriever module first selects a + the work was done during an internship at aws ai lab. question: what is the lightest metal under standard conditions? answer: lithium dpr passage, electrochemical fatigue crack sensor: stainless steel has a density of 8000 kg/m 3 and aluminum alloy has a density of 2700 kg/m 3 . efs detects growing cracks in steel, aluminum, titanium alloys … dhr passage, iron: because of the softness of iron, it is easier to work with than its heavier congeners. the form of iron that is stable under standard conditions … hhr passage, lithium: lithium is a chemical element with symbol li. it is a soft alkali metal. under standard conditions, it is the lightest metal and the lightest solid … candidate set of relevant contexts from a diversified large corpus such as wikipedia; afterwards, a reader module consumes the retrieved evidence to predict an answer. here, retrieval performance is crucial to the accuracy of the qa system as it determines whether the correct context to answer the question can be presented to the reader. while most work in information retrieval focus on document retrieval @xcite , existing work in odqa often splits documents into short passages and directly retrieve passages for the reader @xcite to accommodate reader models that handle shorter sequences most effectively. a drawback of such single-stage passage retrieval approaches is that they tend to be susceptible to distracting passages that contain seemingly relevant local context but not the correct answer, since they cannot incorporate information from other parts of the document (see figure 1 ). further, the large number of passage candidates also contributes negatively to system throughput. to mitigate these issues, @xcite recently proposed a two-stage hierarchical retrieval framework where the retriever first retrieves relevant documents, then discern relevant passages within re-figure 2 : an overview of hhr. it consists of a document retrieval followed by a passage retrieval. each stage uses one of three types of retrievers -sparse, dense, and combined, leading to a total of 9 configurations. despite its success, liu et al.'s ( 2021 ) approach (dense hierarchical retrieval, dhr) relies on dense neural retrievers @xcite for both document retrieval and passage retrieval, which suffers from two key weaknesses. first, neural encoders used in retrieval are often limited in context length for effectiveness and efficiency, which is too short to encompass most documents. as a result, dhr needs to make use of the structure of wikipedia documents, and represents the documents succinctly with title, abstract and table of contents, which is not always available for non-wikipedia text. second, dense retrievers have been shown to suffer from poor generalization on out-of-domain data @xcite , whereas sparse retrievers like bm25 @xcite excel with lexical matches @xcite . in this work, we propose a hybrid hierarchical retrieval (hhr) framework to alleviate these issues. specifically, we investigate the tradeoffs and complementary strengths of sparse retrievers and dense retrievers at both the document retrieval and passage retrieval stages for odqa (see figure 2 ). we find, among other things, that sparse retrievers can complement dense retrievers at both retrieval stages with a simple approach to aggregate results from both. besides in-domain evaluation on the dataset that neural models are trained on, we also perform zero-shot evaluation on unseen datasets to compare the generalization of these retriever architectures. we find that sparse retrievers can help hhr generalize better to unseen data and potentially replace dense retrievers in document retrieval. in addition, we also study the accuracy, storage cost, and latency tradeoff for these architectures under the hhr framework, and offer practical insights to real-world odqa systems that often need to factor these into consideration. our main contributions are as follows. first, we propose a hybrid hierarchical retrieval framework on odqa, and extensively study tradeoffs and complementary strengths of sparse and dense retrievers in both document and passage retrieval. second, we perform both in-domain and out-ofdomain evaluation to provide insight into the generalization performance of different model choices. finally, we present the accuracy-storage-latency landscape for hhr architectures and offer practical insights to real-world applications. passage retrieval. since most reader models in odqa systems struggle to effectively handle long contexts, odqa retrieval is often performed at the passage level (usually around 100 words long). earlier work @xcite relied on bag-of-words-based sparse retrievers such as bm25 @xcite . more recent work showed that neural retrievers can generate effective dense representations for retrieval when trained on odqa @xcite @xcite . @xcite showed, however, that these dense retrievers tend to generalize worse to unseen entities during training since they lack the capacity for lexical matching, which is a strong suit for sparse retrievers and important for out-of-domain generalization. hierarchical retrieval. passage retrievers are limited by the context available in each passage, and can retrieve spurious passages to hurt answer performance. a remedy is to incorporate documentlevel relevancy during passage retrieval. @xcite explored combining document and passage relevancy scores in a bm25 retriever for odqa. @xcite applied this idea to dense retrievers with a hierarchical retrieval framework (dhr), where a document retriever first retrieves documents of high relevancy, followed by a passage retriever to rerank passages within those documents, and our work extends this approach. ## methodology our hybrid hierarchical retrieval (hhr) framework extends dhr, a hierarchical retriever built on dense retrievers that first retrieves top-k d documents and then top-k p passages from those documents. we follow dhr to build the dense retrievers in hhr, and expand both document and passage retrievers to work with sparse retrievers to address limitations of the dhr approach (figure 2 ). specifically, in dhr, to make documents amenable to neural encoders with limited context length, the authors proposed to leverage the document structure of wikipedia articles to construct a document summary that contains the document abstract and table of contents. while effective, this also potentially limits the applicability of this approach to corpora where this information is not available. in contrast, a sparse retriever can easily handle documents of arbitrary lengths efficiently without the need of structure information. besides, dense retrievers tend to generalize poorly to out of domain data. we extend each of the document retrieval and passage retrieval stages with the option to use sparse retrievers to help alleviate this issue, and to help us understand the tradeoff between the two. besides switching between sparse and dense retrievers in hhr, we also introduce a simple heuristic to combine results from both retrievers at the same stage by simply interleaving their top-k/2 results for top-k retrieval to better understand the complementary strengths of sparse and dense retrievers. this yields a total of 9 possible configurations for hhr for our extensive studies. finally, for both sparse and dense passage retrievers in hhr, we implement on-the-fly passage reranking for all passages in the top retrieved documents with pre-computed passage representations. this helps reduce the latency of the passage retrievers in our implementation and provide more realistic insights into the accuracy-storage-latency tradeoff of different hhr settings in real-world systems. ## experimental setup data three commonly used odqa datasets are considered: natural questions (nq) @xcite , web questions (webq) @xcite , and triviaqa @xcite . following liu et al. (2021) , we use the wikipedia dump from dec. 20, 2018 as the source corpus. when splitting documents into passages, the text under same sections are split into non-overlapping passages with a maximum of 100 words. ## conclusion in this paper, we study a hybrid hierarchical retrieval (hhr) framework for odqa that integrates sparse and dense retrievers through a simple aggregation strategy in document and passage retrievers. we demonstrate that sparse retrievers complement dense retrievers in-domain and greatly improve the poor generalization from dense encoders to zeroshot data. our framework addresses the limitation of dhr by achieving better zero-shot performance without relying on document structure information. we also study the accuracy-storage-latency landscape. we believe these findings are critical to the real-world adoption of hhr to odqa systems.