Large Language Models are Built-in Autoregressive Search Engines (2023)

Abstract: Document retrieval is a key stage of standard Web search engines. Existing dual-encoder dense retrievers obtain representations for questions and documents independently, allowing for only shallow interactions between them. To overcome this limitation, recent autoregressive search engines replace the dual-encoder architecture by directly generating identifiers for relevant documents in the candidate pool. However, the training cost of such autoregressive search engines rises sharply as the number of candidate documents increases. In this paper, we find that large language models (LLMs) can follow human instructions to directly generate URLs for document retrieval. Surprisingly, when providing a few Query-URL pairs as in-context demonstrations, LLMs can generate Web URLs where nearly 90% of the corresponding documents contain correct answers to open-domain questions. In this way, LLMs can be thought of as built-in search engines, since they have not been explicitly trained to map questions to document identifiers. Experiments demonstrate that our method can consistently achieve better retrieval performance than existing retrieval approaches by a significant margin on three open-domain question answering benchmarks, under both zero and few-shot settings. The code for this work can be found at https://github.com/Ziems/llm-url .

## introduction along with the success of deep learning, dualencoder based retrievers have become the dominant method for web searching @xcite . for example, dpr @xcite employs two independent encoders to encode the question and the document respectively, then estimates their relevance by computing a single similarity score between two representations. however, these methods suffer from two major drawbacks. first, the representations of questions and documents are typically obtained independently in modern dual-encoder dense retrieval models @xcite , allowing for only shallow interactions between them @xcite . second, the question or document representation is embedded into a single dense vector, potentially missing fine-grained information when computing the similarity between the two vector representations @xcite . instead of computing similarity between question and document embeddings, autoregressive search engines aim to directly generate document identifiers then map them to complete documents in the predetermined candidate pool. this approach has attracted increasing interest in information retrieval (ir) and related fields @xcite @xcite . compared to dual-encoder dense retrieval methods, autoregressive search engines enjoy a number of advantages. first, autoregressive generation models produce document identifiers by performing deep token-level cross-attention, resulting in a better esti-mation than shallow interactions in dense retrievers. second, autoregressive search engines have been shown to have strong generalization abilities, outperforming bm25 in a zero-shot setting @xcite . while it is theoretically possible to scale an autoregressive search engine to the size of a large language model (llm), such as gpt-3 with 175b parameters, in practice it is not feasible due to the computational overhead of training such a large autoregressive search engine from scratch @xcite . to reduce the high training cost of autoregressive search engine, a smaller model size is preferred. however, the results of our pilot study in figure 1 show smaller language models are significantly worse at mapping passages to document identifiers than larger ones. moreover, different retrieval tasks can have unique retrieval requirements. one task may require a model to retrieve factual evidence to support or refute a claim (i.e., fact checking) @xcite while another may require a model to retrieve specific trivia information about an entity (i.e., entity linking) @xcite . it would be better if the retriever was capable of generalizing to new retrieval tasks with only a few examples. in this work, we explore the use of in-context demonstrations to prompt llms to directly generate web urls for document retrieval, namely llm-url. surprisingly, we find that by providing a few (query, url) pairs as contextual demonstrations, large language models (e.g. gpt-3) generate web urls where nearly 90% of the corresponding documents contain answers to opendomain questions. in this way, llms can be thought of as built-in search engines, as they have not been explicitly trained to map questions or documents to identifiers. instead of using newlycreated document identifiers, llm-url leverages existing and widely used document identifiers directly, i.e., urls. we compare our approach to existing document retrieval methods on three different open-domain question answering (qa) datasets: webq @xcite , nq @xcite , and triviaqa @xcite . further, to avoid exceeding the limit on the number of input tokens of llms, we employ an unsupervised passage filtering module to remove irrelevant portions of supporting documents. to summarize, our main contributions are as follows: ## proposed method in this section we describe a new method, which we refer to as llm-url, that employs a large language model (llm) to perform effective and efficient web document retrieval for knowledgeintensive nlp tasks such as open-domain question answering (odqa). odqa is a two step process consisting of a retriever and a reader. given a question q, the goal of the retriever is to find the top-n passages p n relevant to answering q. given q and the top-n relevant passages p n , the goal of the reader is to use internal knowledge along with p n to generate a correct answer a to question q. the passage retriever plays an essential role in this process. when p n contains more passages that have the correct answer, the reader has a higher chance of finding it. instead of heavily training a dedicated retriever, our llm-url solves the problem in a different way as shown in figure 2 . given a question q, our llm-url should find a set of relevant passages to p n and give it to the reader. first, it prompts a llm (e.g., gpt-3) to directly generate m urls for q. by default, it uses "which m wikipedia urls would have the answer?" as the instruction which is appended to each input question as the prompt. we also append the beginning of the wikipedia url ( https://en.wikipedia.org/wiki ) to the end of the prompt to encourage the generation of urls and restrict generation to the wikipedia article url format. as llm has the ability of in-context learning, we take this advantage to enable the fewshot setting in the prompt. the prompt described above also includes a series of in-context demonstrations. each demonstration contains a question sampled from the training set following the prompt described above. at the end of each demonstration, m urls which point to gold-labeled documents are listed. in the zero-shot setting, the original prompt is used without any demonstrations. in the few-shot setting, the original prompt appended to a series of d demonstrations (@xmath0. given the prompt, the llm returns a generated sequence of tokens. ideally these tokens would construct a sequence of m separated urls. in practice, the generated sequence often has extra information such as a proposed answer that is unreliable and needs to be filtered. we use a regular expression to extract all urls from the sequence and discard all extra information. this also filters out many urls that are improperly formatted. after extraction, get requests are made using the extracted urls and the contents of each retrieval is used to create a set of fetched documents d f . often, |d f | < m because some of the generated urls do not follow a correct format or do not point to real web pages on the internet. the set of fetched documents d f can be passed directly to a reader if m is a small value or the reader being used can handle many large documents. however, this is usually not the case. often, d f needs to be filtered such that only a small number of the most relevant passages are given to the reader. to do this, our llm-url first breaks each document @xmath1. the passages from each document are collected into a new set, p f . a scoring function is used to quantify the relevance of each passage with respect to the question q, with high values indicating high relevance with respect to q and low scores indicating low relevance. a simple scoring function such as bm25 can be used or a more complex one such as dpr @xcite can. the passages in p f are then sorted from highest to lowest and the top n are kept as p n . finally, p n are given to a reader along with q to generate an answer. advantages of llm-url : existing autoregressive retrieval methods such as dsi and seal use a pre-trained large language model then fine tune it to take questions as input and generate relevant document identifiers as output @xcite . both dsi and seal do extensive experiments on a variety of document identifiers which are generated by a heavily trained language model. examples of these identifiers include unstructured atomic identifiers, naively structured string identifiers, hierarchical document clustering, and others. llm-url instead uses pre-existing document identifiers that exist on the internet: urls. using urls instead of the aforementioned identifiers has multiple advantages. urls often contain words related to the information they link to, allowing for strong association of topics with their urls. for example, the title of each wikipedia page is used in its url, allowing the llm is able to directly generate the url by leveraging semantic information from the question. to validate the importance of urls themselves, we also experiment with prompting the llm to generate wikipedia titles instead of urls and find recall@1 significantly reduces compared to prompting for url generation. we believe this is because the url format itself helps prompt the model for specific information in a specific format. further, the use of urls allows us to simply obtain the evidence document via a http request without any need of training a model or building an index to find the mapping between identifiers and documents. ## experiments in this section, we present and discuss results from our experiments to "directly" demonstrate that our llm-url is a strong retriever and "indirectly" show that it achieves competitive performance on the odqa task against state-of-the-art solutions. large language model: following figure 1 , the large language model we use to generate urls for our experiments is gpt-3 text-davinci-003 with greedy decoding and a temperature of 0. a variety of different prompts are tested for generating urls, but little difference in performance is observed, so we simply use the best performing prompt which is discussed in section 3. ## conclusion and future work in this paper, we explored whether large language models can generate urls prompted by human instructions for document retrieval. surprisingly, we found that by providing a few (query, url) pairs as in-context demonstrations, large language models (e.g. gpt-3) generated web urls where near 90% of the corresponding documents contain correct answers to open-domain questions in webq. furthermore, by breaking the retrieved documents into passages then ranking them with bm25, we showed a significant number of unnecessary passages could be filtered out while retaining high recall, which outperformed baseline methods by a significant margin. there are numerous exciting directions for future work. while a number of broad spectrum retrieval benchmarks such as bier @xcite exist, it remains to be seen whether the few-shot demonstrations shown in this work can be further tuned for specific retrieval tasks. promptagator @xcite shows significant performance improvements can be achieved by tuning prompts in a similar way. further, it remains to be seen whether fine tuning the prompt for each individual question can further improve the retrieval performance. as with promptagator, prior work has shown using clustering to select diverse demonstrations for any given question further improves retrieval performance as well as downstream qa performance.