Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering (2023)

Abstract: We propose EAR, a query Expansion And Reranking approach for improving passage retrieval, with the application to open-domain question answering. EAR first applies a query expansion model to generate a diverse set of queries, and then uses a query reranker to select the ones that could lead to better retrieval results. Motivated by the observation that the best query expansion often is not picked by greedy decoding, EAR trains its reranker to predict the rank orders of the gold passages when issuing the expanded queries to a given retriever. By connecting better the query expansion model and retriever, EAR significantly enhances a traditional sparse retrieval method, BM25. Empirically, EAR improves top-5/20 accuracy by 3-8 and 5-10 points in in-domain and out-of-domain settings, respectively, when compared to a vanilla query expansion model, GAR, and a dense retrieval model, DPR.

## introduction open-domain question answering (qa) @xcite , a task of answering a wide range of factoid questions of diversified domains, is often used to benchmark machine intelligence @xcite and has a direct application to fulfilling user's information need @xcite . to provide faithful answers with provenance, and to easily update knowledge from new documents, passage retrieval, which finds relevant text chunks to given questions, is critical to the success of a qa system. retrieval in early opendomain qa systems @xcite is typically based on term-matching methods, such as bm25 @xcite or tf-idf @xcite . such methods are sometimes called sparse retrievers, as they represent documents and queries with high-dimensional sparse vectors, and can efficiently match keywords with an inverted index and find relevant passages. despite their simplicity, sparse retrievers are limited by their inability to perform semantic matching for relevant passages that have low lexical overlap with the query. lately, dense retrievers @xcite , which represent documents and queries with dense, continuous semantic vectors, have been adopted by modern qa systems. dense retrievers usually outperform their sparse counterparts, especially when there exists enough in-domain training data. however, dense retrievers have certain weaknesses compared to sparse ones, including: 1) being computationally expensive in training and inference, 2) potential information loss when compressing long passages into fixed-dimensional vectors @xcite , which makes it hard to match rare entities exactly @xcite , and 3) difficulty in generalizing to new domains @xcite . as a result, dense retrievers and sparse ones are usually complementary to each other and can be combined to boost performance. recent studies on query expansion, such as gar @xcite , have attempted to improve sparse retrievers by adding relevant contexts to the query using pre-trained language models (plms), which has been shown effective in closing the gap between sparse and dense retrievers. in this paper, we introduce a novel query expansion and reranking approach, ear, which enhances generative query expansion with query reranking. ear first generates a diverse set of expanded queries with query expansion models, and then trains a query reranker to estimate the quality of these queries by directly predicting the rank order of a gold passage, when issuing these queries to a given retriever, such as bm25. at inference time, ear selects the most promising query expansion as predicted by the query reranker and issues it to the same retriever to find relevant documents. ear is motivated by a simple observation-while the greedy decoding output of a query expansion query: where do they grow hops in the us? ## proposed method we illustrate our proposed method, ear, in figure 1 , along with a comparison with the bm25 and gar pipelines. given the original query q, ear first generates a set of query expansions @xmath0, e 2 , ..., e n } using random sampling. we believe that among these n queries, some may achieve very good retrieval performance. thus, we train a reranker model m to re-score all the queries. here we propose two kinds of rerankers: 1) retrieval-independent (ri) reranker, and 2) retrievaldependent (rd) reranker. both rerankers can estimate the quality of a query expansion without using information from answer annotations. ## query reranking vs passage reranking ear shares similarities with passage reranking (pr). ear reranks the queries before retrieving the passages, while pr reranks the retrieved list of passages after the retrieval process is completed. to better understand the relationship between ear and pr, we implement a bert-based passage reranker, following the method outlined in @xcite , to rerank the retrieval results of gar. the implementation details can be found in appendix c. from the experiments we aim to answer three questions: 1) is ear better than pr? 2) are the contributions of ear and pr model natural questions triviaqa top-5 top-20 top-100 top-5 top-20 top-100 bm25 43.8 62.9 78.3 67.7 77.3 83.9 gar 60.8 73.9 84.7 71.8 79.5 85.3 gar + passage rerank (@xmath1.8 75.7 84.7 77.6 81.0 85.3 gar + passage rerank (@xmath2.7 80.2 84.7 79.2 83.3 85.3 ear-rd 69.3 78.6 86.5 77.6 82.1 86.4 ear-rd + passage rerank (@xmath3.7 82.1 86.5 80.6 84.5 86.4 are ear and pr complementary? we found that the effects of ear-rd and pr can be effectively combined for even better performance. when applying pr on the retrieval results of ear-rd (shown in row 6), we see a significant improvement compared to both row 4 and row 5. this suggests that the contributions of ear-rd and pr are complementary: ear strengthens first-pass retrieval by selecting good queries, while pr rescores all the retrieved passages and generates an entirely new order for these passages. the distinction between these two mechanisms makes improvements accumulative and leads to superior results. extra advantages of ear? an advantage of ear is that it improves retrieval results beyond the top-k passages. in row 4, the top-100 accuracy cannot be improved by pr as it reranks within the top-100 passages. in contrast, the improvements provided by ear are not limited to top-100 passages. as long as ear selects good query expansions, it can improve the whole list of retrieved passages; we can see ear-rd improves the top-100 accuracy of gar from 84.7 to 86.5. ## discussions 6.1 generating training examples with gar in section 3.3, we discussed two methods to construct training examples for ear. in our main experiments, we used t0-3b to randomly sample diverse query expansions. an alternative method was also explored, where we trained @xmath4, then randomly sampled from the hold-out sets. the performance of this method, as shown in ## computational cost and latency we report the latency of dpr, gar, and ear in dense retrieval we first generate dpr document embeddings on 4 gpus for ∼3.5 hours on 21m documents. standard indexing takes ∼10 minutes with a 64gb index size. indexing with the more advanced hierarchical navigable small world (hnsw) (malkov and yashunin, 2018) takes ∼5 hours and results in a huge index size of 142gb. for retrieval, standard indexing takes 22.3s per query, while the highly optimized hnsw can shorten it to 0.04s per query. sparse retrieval for bm25 with pyserini, indexing only takes 0.5 hours, with a very small index size of 2.4gb. retrieval for bm25 takes 0.15s per query. for gar, it needs an extra 0.58s to generate the query expansions, and retrieval time is 0.56s. for ear, it needs 1.29s to batch sample 50 query expansions. ear-ri only takes 0.04s to rerank queries. ear-rd needs extra time to retrieve the top-1 passages for each expansion, which takes an extra 0.70s, and then run the actual reranking process, taking 0.14s, giving a total of 0.84s for query reranking. for retrieval, the time needed for both ear-ri and ear-rd is similar to gar. to conclude, ear inherits the advantage of bm25: fast indexing time and small index size. this makes it possible to index large collections of documents in a relatively short amount of time, which is important for tasks where documents are frequently added or updated. the main cost for ear is the time for sampling query expansions. however, this can potentially be reduced by speedup toolkits that optimize the inference time of transformers, such as fastertransformer foot_2 (3.8∼13× speedup for decoding) or fastseq @xcite 7.7× speedup for bart decoding). moreover, we can leverage model distillation @xcite and quantization @xcite for transformers. we leave these directions for future work. query expansion and reformulation traditionally, query expansion methods based on pseudo relevance feedback utilize relevant context without external resources to expand queries @xcite @xcite . recent studies attempt to reformulate queries using generative models, relying on external resources such as search sessions @xcite or conversational contexts @xcite , or involve sample-inefficient reinforcement learning training @xcite . more recently, gar @xcite explored the use plms for query expansion instead of external resources. a concurrent study @xcite generates multiple expansions with beam search and filters and fuses the results, but ear is aware of the bm25 retriever and could select more promising query expansions and run fewer bm25 retrievals. retrieval for openqa sparse retrieval with lexical features such as bm25 was first explored for openqa @xcite . dense retrieval methods were shown to outperform sparse methods @xcite , while requiring large amounts of annotated data and much more compute. although powerful, dense retrievers often fall short in the scenarios of 1) requiring lexically exact matching for rare entities @xcite and 2) out-of-domain generalization @xcite . for 1), luan et al. (2021) proposed a sparse-dense hybrid model, and chen et al. (2021) trained a dense retriever to imitate a sparse one. for 2), ram et al. (2022) created a pre-training task for dense retrievers to improve zero-shot retrieval and out-of-domain generalization. another recent line of research explores passage reranking with plms to improve performance for both sparse and dense methods. nogueira and cho (2019) first explored bert-based supervised rerankers for standard retrieval tasks and mao et al. (2021b) proposed reranking by reader predictions without any training. sachan et al. (2022) attempt to use an llm directly as the reranker, but it requires huge amounts of computation at inference time and underperforms fine-tuned rerankers. ## conclusion we propose ear, which couples gar and bm25 together with a query reranker to unlock the potential of sparse retrievers. ear significantly outperforms dpr while inheriting the advantage of bm25: fast indexing time and small index size compared to the compute-heavy dpr. cross-dataset evaluation also shows that ear is very good at generalizing to out-of-domain examples. furthermore, we demonstrate that contributions of ear and passage reranking are complementary, and using both methods together leads to superior results. overall, ear is a promising alternative to existing dense retrieval models, providing a new way to achieve high performance with less computing resources.