TOME : A Two-stage Approach for Model-based Retrieval (2023)

Abstract: Recently, model-based retrieval has emerged as a new paradigm in text retrieval that discards the index in the traditional retrieval model and instead memorizes the candidate corpora using model parameters. This design employs a sequence-to-sequence paradigm to generate document identifiers, which enables the complete capture of the relevance between queries and documents and simplifies the classic index-retrieval-rerank pipeline. Despite its attractive qualities, there remain several major challenges in model-based retrieval, including the discrepancy between pre-training and fine-tuning, and the discrepancy between training and inference. To deal with the above challenges, we propose a novel two-stage model-based retrieval approach called TOME, which makes two major technical contributions, including the utilization of tokenized URLs as identifiers and the design of a two-stage generation architecture. We also propose a number of training strategies to deal with the training difficulty as the corpus size increases. Extensive experiments and analysis on MS MARCO and Natural Questions demonstrate the effectiveness of our proposed approach, and we investigate the scaling laws of TOME by examining various influencing factors.

## introduction information retrieval systems have undergone continuous development over the past few decades, with the aim of obtaining relevant resources, such as documents, in response to a user query from a vast collection. with the recent success of pretrained language models (plms) @xcite @xcite , researchers have developed plm-based dense retrievers @xcite , which utilize dual-encoders and nearest neighbor search index for retrieval and achieve significant improvements over sparse retrievers. more recently, a new retrieval paradigm, known as model-based retrieval @xcite , has been introduced by developing an alternative architecture for retrieval. in contrast to traditional retrieval methods, it does not explicitly maintain a corpus index, thereby simplifying the classic index-retrieve-rerank process. typically, a model-based retrieval system is built based on a sequence-to-sequence generation model with an encoder-decoder architecture, such as t5 @xcite and bart @xcite . it accepts a query as input and directly generates the corresponding document identifier via the generation model. despite its attractive benefits in simplifying the retrieval pipeline, model-based retrieval still faces following major challenges. passage generation model when did nevada become a state query here's some history on that: nevada became a state (admitted to the union) october 31, 1864, so ...... https://en.wikipedia . org/wiki/nevada passage url url generation model passage generation stage url generation stage ing inference, resulting in a gap between the training and inference stages. to this end, in this paper, we propose a novel two-stage model-based retrieval approach, tome (as illustrated in figure 1 ), which makes two major technical contributions. moreover, we discover that optimizing modelbased retrieval becomes a challenging task when dealing with a vast corpus. as a result, we propose a number of improved training strategies to optimize the generation models, including query augmentation, passage length reduction, and model scaling. to verify the effectiveness of tome, we conduct extensive experiments on the publicly available ms marco and nq datasets. experimental results demonstrate the effectiveness of the proposed method, including the url identifier design and the two-stage generation process. additionally, case studies indicate that the second stage can tolerate errors induced by the first stage. furthermore, we investigate the scaling laws of tome by examining different model sizes, corpus sizes, and text lengths. we anticipate that these experimental results will facilitate further research on model-based retrieval. model-based retrieval. both sparse retrieval and dense retrieval rely on explicit indices. recently, researchers have proposed model-based retrieval (a.k.a., generative retrieval) models @xcite . these methods consider model parameters as retrieval indices and directly generate the identifiers of related documents. such an idea is initially proposed for entity retrieval @xcite , which autoregressively generates unique entity identifiers. following this approach, researchers have introduced sequenceto-sequence encoder-decoder architecture for document retrieval @xcite @xcite @xcite @xcite . as discussed in the previous section, there still remain issues with model-based retrieval, including the discrepancy between pre-training and fine-tuning, and the discrepancy between training and inference. our work tries to deal with these issues with a two-stage generation architecture with url identifiers. ## approach in this section, we first introduce the task formulation, followed by the description of the proposed two-stage generation approach tome. ## experimental settings this section describes the major experimental settings, including datasets, evaluation metrics, baselines and implementation details. ## experimental results and analysis in this section, we report the experimental results of our proposed approach and conduct comprehensive empirical analysis. ## conclusion in this paper, we introduce tome, a innovative two-stage model-based retrieval approach. to implement our approach, we make two major technical contributions in the design of the identifier and the architecture of two-stage generation. moreover, we also employ a number of training strategies to better optimize our proposed architecture, especially on large-scale corpora. extensive results demonstrate the effectiveness of tome. furthermore, we perform a thorough analysis and summarize the scaling law for the proposed method. we believe such an idea itself is worthwhile for exploring in designing new model-based retrieval architecture.