Query2doc: Query Expansion with Large Language Models (2023)

Abstract: This paper introduces a simple yet effective query expansion approach, denoted as query2doc, to improve both sparse and dense retrieval systems. The proposed method first generates pseudo-documents by few-shot prompting large language models (LLMs), and then expands the query with generated pseudo documents. LLMs are trained on web-scale text corpora and are adept at knowledge memorization. The pseudo-documents from LLMs often contain highly relevant information that can aid in query disambiguation and guide the retrievers. Experimental results demonstrate that query2doc boosts the performance of BM25 by 3% to 15% on ad-hoc IR datasets, such as MS-MARCO and TREC DL, without any model fine-tuning. Furthermore, our method also benefits state-of-the-art dense retrievers in terms of both in-domain and out-of-domain results.

## introduction information retrieval (ir) aims to locate relevant documents from a large corpus given a user issued query. it is a core component in modern search engines and researchers have invested for decades in this field. there are two mainstream paradigms for ir: lexical-based sparse retrieval, such as bm25, and embedding-based dense retrieval @xcite . although dense retrievers perform better when large amounts of labeled data are available @xcite , bm25 remains competitive on out-ofdomain datasets @xcite . query expansion @xcite ) is a long-standing technique that rewrites the query based on pseudo-relevance feedback or external knowledge sources such as wordnet. for sparse retrieval, it can help bridge the lexical gap between the query and the documents. however, query expansion methods like rm3 @xcite have only shown limited success on popular datasets @xcite , and most state-ofthe-art dense retrievers do not adopt this technique. in the meantime, document expansion methods like doc2query @xcite have proven to be effective for sparse retrieval. in this paper, we demonstrate the effectiveness of llms @xcite as query expansion models by generating pseudo-documents conditioned on few-shot prompts. given that search queries are often short, ambiguous, or lack necessary background information, llms can provide relevant information to guide retrieval systems, as they memorize an enormous amount of knowledge and language patterns by pre-training on trillions of tokens. our proposed method, called query2doc, generates pseudo-documents by few-shot prompting llms and concatenates them with the original query to form a new query. this method is simple to implement and does not require any changes in training pipelines or model architectures, making it orthogonal to the progress in the field of llms and information retrieval. future methods can easily build upon our query expansion framework. for in-domain evaluation, we adopt the ms-marco passage ranking @xcite , trec dl 2019 and 2020 datasets. pseudodocuments are generated by prompting an improved version of gpt-3 text-davinci-003 from openai @xcite . results show that query2doc substantially improves the off-theshelf bm25 algorithm without fine-tuning any model, particularly for hard queries from the trec dl track. strong dense retrievers, including dpr @xcite , simlm @xcite , and e5 @xcite also benefit from query2doc, although the gains tend to be diminishing when distilling from a strong cross-encoder based re-ranker. experiments in zero-shot ood settings demonstrate that our method outperforms strong baselines on most datasets. further analysis also reveals the importance of model scales: query2doc works best when combined with the most capable llms while small language models only provide marginal improvements over baselines. to aid reproduction, we release all the generations from text-davinci-003 at https://huggingface.co/datasets/ intfloat/query2doc_msmarco . ## method write a passage that answers the given query: given a query q, we employ few-shot prompting to generate a pseudo-document d as depicted in figure 1 . the prompt comprises a brief instruction "write a passage that answers the given query:" and k labeled pairs randomly sampled from a training set. we use @xmath0. subsequently, we rewrite q to a new query q + by concatenating with the pseudo-document d . there are slight differences in the concatenation operation for sparse and dense retrievers, which we elaborate on in the following section. sparse retrieval since the query q is typically much shorter than pseudo-documents, to balance the relative weights of the query and the pseudodocument, we boost the query term weights by repeating the query n times before concatenating with the pseudo-document d : here, "concat" denotes the string concatenation function. q + is used as the new query for bm25 retrieval. we find that @xmath1. dense retrieval the new query q + is a simple concatenation of the original query q and the pseudo-document d separated by @xcite : for training dense retrievers, several factors can influence the final performance, such as hard negative mining @xcite , intermediate pretraining @xcite , and knowledge distillation from a cross-encoder based re-ranker @xcite . in this paper, we investigate two settings to gain a more comprehensive understanding of our method. the first setting is training dpr @xcite ) models initialized from bert base with bm25 hard negatives only. the optimization objective is a standard contrastive loss: the second setting is to build upon state-of-theart dense retrievers and use kl divergence to distill from a cross-encoder teacher model. ## analysis scaling up llms is critical for our proposed method, a question that naturally arises is: how does the model scale affect the quality of query expansion? ## conclusion this paper presents a simple method query2doc to leverage llms for query expansion. it first prompts llms with few-shot examples to generate pseudo-documents and then integrates with existing sparse or dense retrievers by augmenting queries with generated pseudo-documents. the underlying motivation is to distill the llms through prompting. despite its simplicity, empirical evaluations demonstrate consistent improvements across various retrieval models and datasets.