Synergistic Interplay between Search and Large Language Models for Information Retrieval (2024)

Abstract: Information retrieval (IR) plays a crucial role in locating relevant resources from vast amounts of data, and its applications have evolved from traditional knowledge bases to modern retrieval models (RMs). The emergence of large language models (LLMs) has further revolutionized the IR field by enabling users to interact with search systems in natural languages. In this paper, we explore the advantages and disadvantages of LLMs and RMs, highlighting their respective strengths in understanding user-issued queries and retrieving up-to-date information. To leverage the benefits of both paradigms while circumventing their limitations, we propose **InteR**, a novel framework that facilitates information refinement through synergy between RMs and LLMs. InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections and enables LLMs to enhance prompt formulation using retrieved documents. This iterative refinement process augments the inputs of RMs and LLMs, leading to more accurate retrieval. Experiments on large-scale retrieval benchmarks involving web search and low-resource retrieval tasks show that InteR achieves overall superior **zero-shot** retrieval performance compared to state-of-the-art methods, even those using relevance judgment. Source code is available at https://github.com/Cyril-JZ/InteR.

## introduction information retrieval (ir) is an indispensable technique for locating relevant resources in a vast sea of data given ad-hoc queries @xcite . it is a core component in knowledge-intensive tasks such as question answering @xcite , entity linking @xcite and fact verification @xcite . over the years, the techniques of ir have evolved significantly: from the traditional knowledge base (kb) @xcite to modern search engines (ses) based on neural representation learning @xcite , ir has become increasingly important in our digital world. more recently, the emergence of cutting-edge large language models (llms; e.g., @xcite , @xcite , bard @xcite , llama @xcite ) has further revolutionized the nlp community and given intriguing insights into ir applications as users can now interact with search systems in natural languages. over the decades, search engines like google or bing have become a staple for people looking to retrieve information on a variety of topics, allowing users to quickly sift through millions of documents to find the information they need by providing keywords or a query. spurred by advancements in scale, llms have now exhibited the ability to undertake a variety of nlp tasks in a zero-shot scenario @xcite by following instructions @xcite @xcite @xcite . therefore, they could serve as an alternative option for people to obtain information directly by posing a question or query in natural languages @xcite , instead of relying on specific keywords. for example, suppose a student is looking to write a research paper on the history of jazz music. they could type in keywords such as "history of jazz" or "jazz pioneers" to retrieve relevant articles and sources. however, with llms, this student could pose a question like "who were the key pioneers of jazz music, and how did they influence the genre?" the llms could then generate a summary of the relevant information and sources, potentially sav-ing time and effort in sifting through search results. as with most things in life, there are two sides to every coin. both ir technologies come with their own unique set of advantages and disadvantages. llms excel in understanding the context and meaning behind user-issued textual queries @xcite @xcite , allowing for more precise retrieval of information, while rms expect well-designed precise keywords to deliver relevant results. moreover, llms have the capacity to directly generate specific answers to questions @xcite , rather than merely present a list of relevant documents, setting them apart from rms. however, it is important to note that rms still have significant advantages over llms. for instance, rms can index a vast number of up-to-date documents @xcite , whereas llms can only generate information that falls within the time-scope of the data they were trained on, potentially leading to hallucinated results @xcite @xcite . additionally, rms can conduct quick and efficient searches through a vast amount of information on the internet, making them an ideal choice for finding a wide range of data. ultimately, both paradigms have their own unique set of irreplaceable advantages, making them useful in their respective areas of application. to enhance ir by leveraging the benefits of rms and llms while circumventing their limitations, we consider bridging retrieval augmented generation (rag) and llm-augmented retrieval. fortunately, we observe that textual information refinement can be performed between two counterparts and boost each other. on the one hand, rms can gather potential documents with valuable information, serving as demonstrations for llms. on the other hand, llms generate concise summaries using well-crafted prompts, expanding the initial query and improving search accuracy. to this end, we introduce inter, a novel framework that facilitates information refinement through synergy between rms and llms. precisely, the rm part of inter receives the knowledge collection from the llm part to refine and expand the information in the query. while the llm part involves the retrieved documents from the rm part as demonstrations to enrich the information in prompt formulation. this two-step refinement procedure can be seamlessly repeated to augment the inputs of rm and llm. implicitly, we assume that the outputs of both components supplement each other, leading to more accurate retrieval. we evaluate inter on public large-scale retrieval benchmarks involving web search and lowresource retrieval tasks following prior work @xcite . the experimental results show that inter can conduct zero-shot retrieval with overall better performance than state-of-the-art methods, even those using relevance judgment foot_0 , and achieves new state-of-the-art zero-shot retrieval performance. overall, our main contributions can be summarized as follows: zero-shot dense retrieval many prior works consider training dense retrieval models on highresource passage retrieval datasets like natural questions (nq) @xcite (133k training examples) or ms-marco @xcite (533k training examples) and then evaluating on queries from new tasks. these systems @xcite are utilized in a transfer learning configuration @xcite . however, on the one hand, it is time-consuming and expensive to collect such a vast training corpus. on the other hand, even ms-marco has limitations on commercial use and cannot be used in a wide range of real-world applications. to this end, recent work @xcite proposes building zero-shot dense retrieval systems that require no relevance supervision (i.e., relevance label between a pair of query and document), which is considered "unsupervised" as the only supervision resides in the llm where learning to follow instructions is conducted @xcite . in this work, we follow this zero-shot unsupervised setting and conduct information refinement through synergy between rms and llms without any relevance supervision to handle the above issues. enhance retrieval through lms recent works have investigated using auto-regressive language models to generate intermediate targets for better retrieval @xcite while identifier strings still need to be created. other works consider "retrieving" the knowledge stored in the parameters of pre-trained language models by directly generating text @xcite . some researchers @xcite @xcite utilize lm to expand the query and incorporate these pseudo-queries for enhanced retrieval while others choose to expand the document @xcite . besides, lms can also be exploited to provide references for retrieval targets. for instance, genread @xcite directly generates contextual documents for given questions. enhance lms through retrieval on the contrary, retrieval-enhanced lms have also received significant attention. some approaches enhance the accuracy of predicting the distribution of the next word during training @xcite or inference @xcite through retrieving the k-most similar training contexts. alternative methods utilize retrieved documents to provide supplementary context in generation tasks @xcite @xcite . webgpt @xcite further adopts imitation learning and uses human feedback in a textbased web-browsing environment to enhance the lms. llm-augmentor @xcite improves llms with external knowledge and automated feedback. replug @xcite prepends retrieved documents to the input for the frozen lm and treats lm as a black box. demonstrate-search-predict (dsp) @xcite obtains performance gains by relying on passing natural language texts in sophisticated pipelines between a lm and a rm, which is most closely related to our approach. however, they rely on composing two parts with in-context learning and target on multi-hop qa. while we aim at conducting information refinement via multiple interactions between rms and llms for large-scale retrieval. then the top-k documents, denoted as d that have the highest similarity scores when compared with query retrieval retrieval generate generate query expansion prompt formulation query query query rm llm rm outputs llm outputs the query q, are retrieved by rms regardless of whether the retrieval is sparse or dense. noting that as for dense retrieval, following existing methods @xcite , we pre-compute each document's vector v d for efficient retrieval and build the faiss index @xcite over these vectors, and use contriever @xcite as the backbone of query encoder e q and document encoder e d . generative retrieval: the llm part generative search is a new paradigm of ir that employs neural generative models as search indices @xcite @xcite . recent studies propose that llms further trained to follow instructions could zero-shot generalize to diverse unseen instructions @xcite @xcite . therefore, we prepare textual prompts p that include instructions for the desired behavior to q and obtain a refined query q ′ . then the llms g such as chatgpt (openai, 2022) take in q ′ and generate related knowledge passage s. this process can be illustrated as follows: ## inter on top of the preliminaries, we introduce inter, a novel ir framework that iteratively performs information refinement through synergy between rms and llms. the overview is shown in figure 1 . during each iteration, the rm part and llm part refine their information in the query through interplay with knowledge collection (via llms) or retrieved documents (via rms) from previous iteration. specifically, in rm part, inter refines the information stored in query q with knowledge collection s generated by llm for better document retrieval. while in llm part, inter refines the information in original query q with retrieved document d from rm for better invoking llm to generate most relevant knowledge. this twostep procedure can be repeated multiple times in an iterative refinement style. ## conclusion in this work, we present inter, a novel framework that harnesses the strengths of both large language models (llms) and retrieval models (rms) to enhance information retrieval. by facilitating information refinement through synergy between llms and rms, inter achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods, and even those using relevance judgment, on large-scale retrieval benchmarks involving web search and low-resource retrieval tasks. with its ability to leverage the benefits of both paradigms, inter may present a potential direction for advancing information retrieval systems.