Exploring the Best Practices of Query Expansion with Large Language Models (2024)

Abstract: Large Language Models (LLMs) are foundational in language technologies, particularly in information retrieval (IR). In this paper, we thoroughly explore the best practice of leveraging LLMs for query expansion. To this end, we introduce a training-free, straightforward yet effective framework called Multi-Text Generation Integration (MuGI). This approach leverages LLMs to generate multiple pseudo-references, which are then integrated with the original queries to enhance both sparse and dense retrieval methods. Additionally, we introduce a retrieval pipeline based on MuGI, which combines the strengths of sparse and dense retrievers to achieve superior performance without the need for costly pre-indexing. Our empirical findings reveal that: (1) Increasing the number of samples from LLMs benefits IR systems; (2) A balance between the query and pseudo-documents, and an effective integration strategy, is critical for high performance; (3) Contextual information from LLMs is essential, even boost a 23M model to outperform a 7B baseline model; (4) Pseudo relevance feedback can further calibrate queries for improved performance; and (5) Query expansion is widely applicable and versatile, consistently enhancing models ranging from 23M to 7B parameters. Our code and all generated references are made available at https://github.com/lezhang7/Retrieval_MuGI.

## introduction information retrieval (ir) is crucial for extracting relevant documents from large databases, serving as a key component in search engines, dialogue systems @xcite , questionanswering platforms @xcite @xcite , recommendation systems @xcite , and retrieval augmented generation (rag) frame-* equal contribution work @xcite @xcite . query expansion, a key technique for enhancing ir efficacy @xcite @xcite , traditionally employs pseudo-relevance feedback (prf) @xcite ) from initial retrieval results. however, its effectiveness is constrained by the quality of these coarse results. recently, large language models (llms), such as chatgpt, have demonstrated exceptional capabilities in language understanding, knowledge storage, and reasoning @xcite . motivated by these advancements, some studies have explored leveraging llms for zero-shot query expansion @xcite @xcite . while these methods have shown empirical effectiveness, they also present certain limitations. lamer @xcite generates potential answers by utilizing llms to rewrite bm25 candidates for expansion. however, its performance is highly dependent on the quality of the initial retrieval. both hyde @xcite and query2doc @xcite leverage the knowledge stored in llms. while hyde demonstrates effective performance with contriver, it performs poorly with lexical-based retrievers @xcite . conversely, query2doc is effective with both sparse and dense retrieval methods, but strong models may not benefit as much as weaker ones @xcite . moreover, the integration and balance between pseudo references and queries are under-explored in these studies. to address these limitations, we explore best practices for utilizing query expansion with llms for information retrieval. in this paper, we delve into several specific research questions: rq1: is there a universal query expansion method that effectively serves both lexical-based and neural-based retrievers, applicable to both weak and strong models without prior constraints? rq2: are multiple pseudo-references more beneficial than a single one? rq3: how can the query and pseudoreferences be balanced for lexical-based retrievers? rq4: what is the most effective method for integrating multiple pseudo-references with a query in dense retrievers? we introduce a framework named multi-text generation integration (mugi) to explore these questions. mugi employs a zero-shot approach to generate multiple pseudo-references from llms, integrating them with queries to enhance ir efficiency. our empirical experiments demonstrate that: (1) increasing the number of samples from llms benefits ir systems. (2) mugi demonstrates versatility and effectiveness across both lexical and dense retrievers and models of various sizes. remarkably, it enables a 23m-parameter dense retriever to outperform a larger 7b baseline. ## related work information retrieval focuses on the efficient and effective retrieval of information in response to user queries. best matching 25 (bm25) @xcite advances beyond earlier probabilistic models by incorporating document length normalization and non-linear term frequency scaling, thereby enhancing the alignment of queries with documents. dense retrievers such as dpr @xcite employ deep neural networks to identify semantic relationships between queries and documents by measuring the cosine similarity of their text embeddings. existing efficient ir systems typically use a retrieval & rerank pipeline @xcite @xcite : initially, a retrieval mechanism, such as bm25 or a bi-encoder, identifies a broad set of potentially relevant documents. subsequently, a stronger ranker, usually a cross-encoder, meticulously scores the relevance of these documents, enhancing the precision of the final results. llms for ir the use of llms in ir falls into two primary categories @xcite : finetuning llms as retrieval models and employing them for zero-shot ir. this paper concentrates on zero-shot ir, where typical approaches involve leveraging the reasoning capabilities of llms for direct document ranking @xcite or relevance assessment @xcite . while effective, these methods are limited by llms' input length constraints, making them better suited for the rerank phase. another line of research focuses on using llms to synthesize additional high-quality training datasets to improve existing models @xcite @xcite . other works, such as hyde @xcite , query2doc @xcite , and lamer @xcite , explore query expansion. they leverage llms to create pseudo-references or potential answers, enhancing queries for better retrieval outcomes. mugi is a query expansion framework that leverages llms to enhance queries. unlike previous works, which are limited by inherent constraints, mugi offers broader applicability and versatility as it seamlessly integrates with both lexical and dense retrievers. by utilizing and intergrating a wealth of contextualized information from multiple references, mugi surpasses existing techniques in both in-domain and out-of-distribution evaluations by more effectively capturing essential keywords and enriching the background context. ## method we begin by discussing ir preliminaries and then introduce our mugi framework. ## experiments in this section, we provide answers for rq1 and rq2 in §4.2 and §4.3 and answers for rq3 and rq4 in §4.4. ## conclusion this paper explores best practices for query expansion methods in information retrieval with llms. we present the multi-text generation integration (mugi) framework, a technique that markedly improves information retrieval by integrating the query with multiple generated passages through an adaptive reweighting strategy, feature pooling, and query calibration. our empirical findings show that mugi significantly enhances the performance of both sparse and dense retrieval models. we offer a comprehensive discussion of best practices, informed by thorough experimentation.