Auto Search Indexer for End-to-End Document Retrieval (2023)

Abstract: Generative retrieval, which is a new advanced paradigm for document retrieval, has recently attracted research interests, since it encodes all documents into the model and directly generates the retrieved documents. However, its power is still underutilized since it heavily relies on the “preprocessed” document identifiers (docids), thus limiting its retrieval performance and ability to retrieve new documents. In this paper, we propose a novel fully end-to-end retrieval paradigm. It can not only end-to-end learn the best docids for existing and new documents automatically via a semantic indexing module, but also perform end-to-end document retrieval via an encoder-decoder-based generative model, namely Auto Search Indexer (ASI). Besides, we design a reparameterization mechanism to combine the above two modules into a joint optimization framework. Extensive experimental results demonstrate the superiority of our model over advanced baselines on both public and industrial datasets and also verify the ability to deal with new documents.

## introduction search engines are widely deployed on web applications to meet users' daily information requirements @xcite . given a user query, search engines usually first retrieve candidate documents from a huge document collection and then rank them to return a ranking list. consequently, the performance and efficiency of document retrieval are essential to the final search quality. recently, a new end-to-end document retrieval framework named generative retrieval is proposed to develop a differentiable indexer, which directly maps a given query to the relevant document identifiers (docids) via a seq2seq model @xcite . specifically, some policies are first applied to preprocess all the existing documents for docids such * corresponding author. † the authors contribute equally. as assigning unique integers for documents @xcite . given the preprocessed docids, a transformer-based model is employed to encode the document-docid mapping information into its parameters, and meanwhile is trained to generate relevant docids directly from a given query. as such, by adding a preprocessing phase, it turns the whole index-retrieve process into a generation task. despite the great success of these methods, the power of generative retrieval is still underutilized since they rely on the pre-processed docids, thus leading to the following limitations. (1) new documents cannot be seamlessly retrieved by an existing trained indexer. on the one hand, docids are preprocessed so that new documents cannot obtain their docid assignments directly from the retrieval model. on the other hand, even if their docids are obtained by the same "pre-processing" policy, these new docids are usually unknown semantics to the retrieval model. (2) existing preprocessing policies are confined to one-to-one mapping between documents and docids. accordingly, only one single document can be retrieved for each retrieval calculation. it deviates from the intention of the retrieving-ranking framework, i.e., a groups of relevant documents are expected to be efficiently retrieved in the retrieving stage @xcite . we argue to assign similar documents with same docid, which supports retrieving more documents at the same computational cost. (3) the preprocessing phase is independent of the index-retrieve process. consequently, the caused semantic gap between the docids in preprocessing phase and the embedding space in index-retrieve process limits the performance of generative retrieval. however, it is not trivial to automatically learn the best docids within a joint framework, since the docids, which is served as the generation ground-truths, cannot maintain the gradient flow because they must appear in discrete form by an argmax function. therefore, the docid learning process and the index-retrieve process are still independent of each other even if they are integrated together. in this paper, we propose a novel fully end-toend generative retrieval paradigm, auto search indexer (asi). it combines both the end-to-end learning of docids for existing and new documents and the end-to-end document retrieval into a generative model based joint optimization framework. specifically, we model document retrieval problem as a seq2seq task, i.e., with user queries as input, it outputs the docids of retrieved documents. then, we design a semantic indexing module, which learns to automatically assign docids to existing / new documents. besides, we design two semantic-oriented losses for it, which makes semantically similar documents share the same docids and assigns different docids to dissimilar documents. as such, the new document will be assigned an existing docid based on its content, or a new docid but belonging to the same semantic space as other docids. furthermore, a reparameterization mechanism is proposed to enable gradient to flow backward through the semantic indexing module, thus supporting joint training for all modules. extensive experiments on public and industrial datasets show that the proposed asi outperforms the state-of-the-art (sota) baselines by a significant margin in document retrieval, and demonstrate that our semantic indexing module automatically learns meaningful docids for documents. the contributions are summarized as follows: ## related work studies about document retrieval can be roughly divided into three categories: sparse retrieval, dense retrieval and generative retrieval, which are briefly introduced as follows. ## conclusion in this paper, we make the first attempt to propose a fully end-to-end pipeline, auto search indexer (asi), which supports both end-to-end docid indexing and end-to-end document retrieval within a joint framework. extensive experiments on the public and industrial datasets show that our asi outperforms the sota methods for document retrieval. besides, the experiments also demonstrate the superiority of asi for handling new documents and verify the effectiveness of its docid assignments.