Retrieval-Augmented Retrieval: Large Language Models are Strong Zero-Shot Retriever (2024)

Abstract: We propose a simple method that applies a large language model (LLM) to large-scale retrieval in zero-shot scenarios. Our method, the Large language model as Retriever (LameR), is built upon no other neural models but an LLM in a retrieval-augmented retrieval fashion, while breaking brute-force combinations of retrievers with LLMs and lifting the performance of zero-shot retrieval to be very competitive on benchmark datasets. Essentially, we propose to augment a query with its potential answers by prompting LLMs with a composition of the query and the query’s in-domain candidates. The candidates, regardless of correct or wrong, are obtained by a vanilla retrieval procedure on the target collection. As a part of the prompts, they are likely to help LLM generate more precise answers by pattern imitation or candidate summarization. Even if all the candidates are wrong, the prompts at least make LLM aware of in-collection patterns and genres. Moreover, due to the low performance of a self-supervised retriever, the LLM-based query augmentation becomes less effective as the retriever bottlenecks the whole pipeline. Therefore, we propose to leverage a non-parametric lexicon-based method (e.g., BM25) as the retrieval module to capture query-document overlap in a literal fashion. As such, LameR makes the retrieval procedure transparent to the LLM, thus circumventing the bottleneck.

## introduction large-scale (or first-stage) is to fetch top relevant documents for a given text query from a huge collection with millions to billions of entries. it is indispensable in information-seeking tasks, such as open-domain question answering @xcite , web search @xcite , knowledgegrounded dialogue @xcite , etc. recently, it is also leveraged as a core retrieval-augmenting module to enrich large language models (llms) with up-to-date or domain-specific knowledge @xcite , which reduces the hallucination problem @xcite and improves the faithfulness of generated texts @xcite . thereby, large-scale retrieval is a long-term research problem, attracting research efforts from academia and industry. in the last decade, large-scale retrieval relies heavily on deep representation learning techniques, from bag-of-words (bow) @xcite to pre-trained language models (plms) @xcite . compared to supervised representation learning @xcite that requires labor-intensive annotations on query-document pairs, self-supervised (or zeroshot) learning @xcite @xcite on in-domain pseudo pairs can be readily generalized to any corpora without human-crafted annotations. nonetheless, the zero-shot retriever usually results in an inferior retrieval quality @xcite , even worse than a non-parametric term-based bm25 retrieval @xcite . fortunately, recent surging llms provide a shortcut to reach zero-shot retrieval by augmenting a query with its potential answering elicited from the llms @xcite . coupled with a self-supervised retriever, contriever @xcite , it delivers superior retrieval performance, even surpassing a number of supervisedly finetuned retrievers. but, such a brute-force combination of a self-supervised retriever with a versatile llm leads to a major problem. the answer elicitation is merely based on prompting llms with short, intent-ambiguous, and domain-vague retrieval queries. due to the ambiguity of user queries and unawareness of in-domain corpora, the llms are likely to generate spurious and out-ofdomain answers to the queries @xcite , making the query augmentation even more toxic. to circumvent this issue, we propose a brandnew and simple paradigm for large-scale retrieval, called lamer. essentially, during eliciting llms for answers to a query, we inject the query's top answer candidates into the prompt, where the candidates are obtained by applying a vanilla retrieval procedure to the query. as such, the llms are prone to distinguish and imitate the candidates @xcite , while summarizing or/and re-writing new ones with internal knowledge of the llms. moreover, despite correct or wrong candidates, they can at least provide demonstrations about in-domain patterns and knowledge @xcite @xcite . moreover, though the llms now generate more precise, and reliable query augmentations, the whole pipeline is likely to be bottlenecked by the weak retriever trained on pseudo data in a selfsupervised manner. therefore, we also propose to get rid of any learnable parametric retrievers, while opting for non-parametric term-or lexicon-based retrieval methods (e.g., bm25 in our experiments) in our lamer. in contrast to model-specific compressed and/or latent embeddings from a deep retriever, the lexicon-based retrieval methods capture lexicon overlap between augmented queries and in-collection documents in a literal fashion, thus taking the outputs of llms in a transparent mode and bypassing the performance bottleneck. we evaluate lamer on several benchmark datasets of large-scale retrieval by following @xcite . our results show that our proposed method achieves the best retrieval qualities on most datasets compared to other zero-shot competitors. also, it can surpass the llm-based retriever with in-context labeled demonstrations and outperform the baseline retrievers fine-tuned on full datasets. ## observation in our pilot experiments, we observed the bruteforce combination of a versatile llm with weak retriever leads to certain demerits, which primarily motivates this work. bottleneck by self-supervised retriever. due to the weakness of a self-supervised dense retriever in representing capability, the whole pipeline is bottlenecked by the retriever, even though correct answers are likely to be generated by the strong llm. as illustrated in figure 1 , strengthening llms in qa-style query augmentation (i.e., hyde @xcite , which elicits an llm to generate answers as query augmentation) hardly improves retrieval performance. here, 'd003' and '3.5t' denote text-davinci-003 and gpt-3.5-turbo by openai, respectively. mismatch w/ term-based retriever. due to unawareness of in-domain corpora, llm is likely to generate out-of-domain answers to a given contextshort and intent-vague query, making the query augmentation even toxic. thanks to the fuzzy capability of dense retrievers, such query augmentation still bring remarkable improvement in search quality. however, when it comes to lexicon-based retrieval (say bm25), the improvement will be reduced due to out-of-domain augmentations. quantitatively, as in figure 2 , 'contriever' is a sota self-supervised dense retriever while 'bm25' is a representative lexicon-based retrieval. it is observed that although bm25 can beat contriever in the vanilla setting, hyde brings twice more improvement to contriever than bm25, making bm25 less competitive. ## related work although an llm can directly generate relevant documents and even the final answer for a user query upon its parametric memory, such a generative information-seeking approach is limited by: i) out-of-date corpora are learned in the parametric memory, ii) unreliable, and hallucinative text is frequently generated, and iii) the domain of generated text cannot be specified as demand. in contrast, information retrieval aims to provide in-domain and reliable documents relevant document collection 𝔻𝔻 bm25 system formulate qa prompt difference between a mcdouble and a double cheeseburger a. there is a discernible diff … b. triple cheeseburger from … c. at first glance, the double … 1. the difference between a mcdouble and a double cheeseburger is primarily the amount of cheese .. 2. the main difference between a mcdouble and a double cheeseburger … index docs retrieved candidates ℂ 𝒒𝒒 generated answers 𝔸𝔸 𝒒𝒒 query 𝒒𝒒 aug query " 𝒒𝒒 results $ 𝔻𝔻 ! retrieve retrieve figure 3 : large language model as retriever (lamer). please see therefore, many research efforts have recently been dedicated to applying large language models (llms), such as the gpt series, to information retrieval tasks for superior search performance. the majority of these works are in few-shot or zero-shot scenarios. @xcite proposed a generate-then-read pipeline instead of the traditional retrieve-then-read pipeline. @xcite introduced a few-shot dense retrieval approach for different tasks with different retrieval intents. @xcite proposed a data augmentation method for domain adaptation for open-domain qa, where a document is passed to llm for the generation of its possible queries. @xcite focused on zero-shot dense retrieval using the hypothetical document embedding (hyde) method to generate potential answers by llm as query augmentation. @xcite and @xcite leveraged a fine-tuned ranker (on ms-marco in a supervised manner) to filter llm-generated data for better query-document quality and thus superior performance. @xcite designed a two-stage llm pipeline for zero-shot query generation and reranker-distilled retriever. @xcite utilized a few-shot query-document demonstration to generate documents for a new query as the query's augmentation. unlike these works, we focus on the zero-shot retrieval scenario, and neither conduct any in-domain data augmentation for domain-specific retriever training nor introduce any other retrieval or/and intermediate models except for a frozen llm. please refer to appendix a for more discussions of §a.1 zero-shot large-scale retrieval (which is our scope), §a.2 in-context learning (as we could regard the bm25-retrieved docs as unlabeled demonstrations), and §a.3 retrieval & rerank pipeline (as we have a two-step retrieval pipeline). ## language language model as retriever this section begins with a task definition, followed by elaborations on three components to achieve lamer -non-parametric lexicon-based retriever ( §4.1), candidate-prompted answer generation ( §4.2), and answer-augmented large-scale retriever ( §4.3). lamer is illustrated in figure 3 . task definition: zero-shot large-scale retrieval. providing a huge collection consisting of many documents, @xmath0, the goal of 'largescale retrieval' is to rank the whole d in descending order according to the relevance score between a given text query q and each d i . the relevance score is usually derived by a high-efficient retrieval model that operates on a pre-indexed |d| and an onthe-fly q to satisfy real-time requirements. meantime, 'zero-shot' means that there is no training set with labeled positive query-document pairs for supervised representation learning. ## experiment in this section, we will conduct extensive experimental evaluations of the proposed retrieval method and compare it with strong competitors. implementation of lamer is available at https://github. com/taoshen58/lamer . ## conclusion we propose a retrieval method based merely on an llm and a simple bm25 algorithm, without any dependence on learnable retrieval models. as such, all the operations are performed in the consistent interface of natural language (i.e., language-based query augmentation and lexicon-overlap retrieval relevance), without the performance bottleneck of a fragile self-supervised model-based retriever. our evaluations verify the effectiveness of the proposed lamer, supporting the llm can solely serve as a strong retriever without any in-domain annotated query-document pairs. limitation i) instruction sensitivity: identical to other promptbased llm applications, this work would also be sensitive to the instructions with different llms, which may consume a lot of human effort on prompt writing. ii) computation overheads: as stated in 5.4, although the 2-stage retrieval procedure in lamer is very fast by inheriting bm25, lamer is constrained by calling the llm for answer generation in terms of computation overheads. to overcome these limitations, in the future we will explore specializing in a relatively smaller llm for query-augmentation purposes.