How Does Generative Retrieval Scale to Millions of Passages? (2023)

Abstract: The emerging paradigm of generative retrieval re-frames the classic information retrieval problem into a sequence-to-sequence modeling task, forgoing external indices and encoding an entire document corpus within a single Transformer. Although many different approaches have been proposed to improve the effectiveness of generative retrieval, they have only been evaluated on document corpora on the order of 100K in size. We conduct the first empirical study of generative retrieval techniques across various corpus scales, ultimately scaling up to the entire MS MARCO passage ranking task with a corpus of 8.8M passages and evaluating model sizes up to 11B parameters. We uncover several findings about scaling generative retrieval to millions of passages; notably, the central importance of using synthetic queries as document representations during indexing, the ineffectiveness of existing proposed architecture modifications when accounting for compute cost, and the limits of naively scaling model parameters with respect to retrieval performance. While we find that generative retrieval is competitive with state-of-the-art dual encoders on small corpora, scaling to millions of passages remains an important and unsolved challenge. We believe these findings will be valuable for the community to clarify the current state of generative retrieval, highlight the unique challenges, and inspire new research directions.

## introduction for the last several years, dual encoders @xcite @xcite have dominated the landscape for first-stage information retrieval. they model relevance by mapping queries and documents into the same embedding space, optimized via contrastive learning @xcite . dense embeddings are pre-computed for all documents in a corpus and stored in an external index, enabling fast approximate nearest neighbor search @xcite to retrieve relevant documents. cross-encoders based on large transformer models @xcite @xcite often function on top of these retrieved documents to further refine the top results. recently, the emerging paradigm of generative retrieval @xcite sought to replace this entire process with a single sequence-to-sequence transformer model @xcite , showing promising results against dual encoders given a sufficiently small corpus size. since then, various techniques, such as @xcite @xcite @xcite , have aimed to improve the effectiveness of generative retrieval models, either with alternative document identifier formulations, architecture changes, or training objectives. such work, however, has only evaluated generative retrieval over relatively small corpora on the order of 100k documents, such as natural questions @xcite , trivi-aqa @xcite , or small subsets of the ms marco document ranking task @xcite . despite these research contributions, several open questions remain unanswered, including how well current generative retrieval techniques work on larger corpora and which aspects of generative retrieval models proposed so far are vital as we scale the corpora. in this paper, we conduct the first empirical study of generative retrieval techniques over the entire ms marco passage-level corpus, evaluating its effectiveness over 8.8m passages. we select popular approaches in recent works and evaluate them first on natural questions and triviaqa to establish a definitive ablation of techniques in a con-trolled setup. our experiments mainly focus on evaluating techniques proposed by @xcite , @xcite , and @xcite . we evaluate three main design aspects in our study: we then scale up the corpus size leveraging the ms marco passage ranking task, beginning with a subset of 100k passages before increasing the count to 1m and 8.8m passages (the entire set). incrementally doing so allows us to establish which techniques remain effective as corpus size and difficulty scale. finally, to explore the effect of model scaling on retrieval effectiveness on large corpora, we evaluate a set of techniques with promising results at the t5.1.1-base (220m) scale @xcite and modify the parameterization to consider up to 11b parameters. as the parameter distributions vary between methods, for instance, atomic docids result in additional parameters equal to the embedding dimension times corpus size, while the naive docids do not cost anything beyond the core transformer model, we aim to provide some insight into the trade-off of different parameter allocations on a large corpus. while our experimental findings are nuanced, we summarize the main findings as follows: in the ms marco passage ranking task, we found that a scaled model, trained only on synthetic queries and leveraging naive docids, outperforms all other techniques we considered. on a small subset of 100k passages, a t5-base model trained with this strategy achieves 82.4 mrr@10 (section a.3.1), competitive with gtr-base @xcite ) at 83.2 mrr@10. however, on the setting with 8.8m passages, a t5-xl model trained with this approach achieves only 26.7 mrr@10. the field of generative retrieval is rapidly evolving, yet matching the effectiveness of top-tier dense retrieval models at scale remains a significant, unresolved challenge. our results suggest the need for continued research into generative retrieval and more fundamental advances to the paradigm before we can fully leverage the power of scaling up model parameters. we believe our findings will help the research community better understand the challenges faced while applying generative retrieval models to larger corpora and inspire new research in this field. ## methods we revisit the design details of the generative retrieval method, using the differentiable search index (dsi) @xcite as the baseline, and describe multiple techniques introduced in subsequent works that we ablate and study in this work. for a comprehensive review of related work, we refer the reader to section a.1 in the appendix. 1306 2.1 background dsi @xcite transforms the retrieval task into sequence-to-sequence (seq2seq) modeling, using queries as inputs and relevant document identifiers (docids) as generation targets. the corpus, namely the mapping between the document's content and its identifier, is encoded using the parameters of the llm. dsi achieves this by leveraging two seq2seq tasks: indexing and retrieval. during training, the model learns to generate the docid given the document content (indexing task) or a relevant query (retrieval task). at inference, the model processes a query and generates a ranked list of identifiers as retrieval results. ## experimental setting we limit ourselves to english retrieval tasks, focusing on the behavior of generative retrieval models at varying corpus scales. ## experimental results we report our results in three parts. first, we ablate all the methods from section 2 using t5-base on small-scale datasets: nq100k and triviaqa. we observe which techniques work best on this small scale with widely studied datasets. then, we transfer the same techniques and scale up to the entire ms marco passage ranking dataset. finally, to understand whether we can attribute the effectiveness bump from atomic docids to additional model parameters, we fix the corpus to be msmarcofull and scale the model size up to 11b. across the tables, at. refers to atomic docids, nv. naive docids, and sm. semantic docids. ## analysis & discussion the results of this work raise major questions, including: why are synthetic queries effective? and which model scaling approach is the best? we present an extended analysis, including additional experiments, to provide more insight on these topics in section a.3 of the appendix. finally, we discuss the limitations of our study and future directions for the paradigm in section 7 and a.4, respectively. ## conclusion we provide the first empirical study of generative retrieval methods over the large-scale 8.8m passage corpus of the ms marco passage ranking task. of the various methods from the literature that we consider in this work @xcite @xcite , we find that the use of synthetic queries as a document representation strategy is the only approach that remained effective and necessary, as we scaled up the corpus size. we highlight the importance of accounting for the compute cost of techniques; keeping the parameter count fixed, we find that naive methods outperform more sophisticated ones on the full ms marco dataset. our best result on ms marco passage ranking uses only synthetic queries and naive docids in the target space for the training task, with the model scaled to t5-xl (3b). this model only achieves an mrr@10 of 26.7. surprisingly, increasing parameters for the same setting up to xxl (11b) scores worse. altogether, our results highlight the unique challenges currently facing the generative retrieval paradigm, the need for closer attention to method comparisons, and more fundamental improvements before we can fully leverage the power of larger language models in the future.