Understanding Differential Search Index for Text Retrieval (2023)

Abstract: The Differentiable Search Index (DSI) is a novel information retrieval (IR) framework that utilizes a differentiable function to generate a sorted list of document identifiers in response to a given query. However, due to the black-box nature of the end-to-end neural architecture, it remains to be understood to what extent DSI possesses the basic indexing and retrieval abilities. To mitigate this gap, in this study, we define and examine three important abilities that a functioning IR framework should possess, namely, exclusivity, completeness, and relevance ordering. Our analytical experimentation shows that while DSI demonstrates proficiency in memorizing the unidirectional mapping from pseudo queries to document identifiers, it falls short in distinguishing relevant documents from random ones, thereby negatively impacting its retrieval effectiveness. To address this issue, we propose a multi-task distillation approach to enhance the retrieval quality without altering the structure of the model and successfully endow it with improved indexing abilities. Through experiments conducted on various datasets, we demonstrate that our proposed method outperforms previous DSI baselinesThe code and data for this work can be found at https://github.com/VerdureChen/Understang_DSI .

## introduction recent advancements in the field of information retrieval (ir) have sparked a growing interest in differentiable search index (dsi) @xcite . unlike traditional methods, which involve building an index before retrieval @xcite @xcite , dsi and related techniques such as dsi-qg @xcite and nci @xcite do not rely on external indexes to store data. instead, these methods map user queries directly to the identifiers (ids) of the relevant documents, providing a simpler and more efficient retrieval process. this novel autoregressive approach, represented by dsi, has expanded the potential ir applications due to its ease of use, minimal index storage requirements, and end-to-end retrievability. however, despite the novel retrieval mechanism of dsi, current dsi models still rely on relevance signals of query-passage pairs for training. these models, which map short texts to specific document ids, do not have an explicit interaction between the query and the document during retrieval, unlike dense retrieval models @xcite @xcite @xcite and cross-attention rerankers @xcite @xcite @xcite . this training approach and the inherent properties of the model may lead to two problems. first, due to the lack of explicit modeling of inter-document associations and an explicit querydocument relevance measurement, the model may only learn a unidirectional mapping from short texts to specific ids, without understanding how the document is relevant to the query, leading to somewhat random output in the ranking list. second, to reduce computational complexity, dsi models often simply represent a document by a small number of tokens or pseudo-queries. however, this approach may result in a reduced capacity to differentiate between documents and capture crucial relevant information. this study aims to deepen the understanding of dsi by evaluating its suitability as an end-toend model for indexing and retrieval. to achieve this, dsi-qg @xcite , a recent enhancement of dsi using pseudo queries for the model training, is used as a representative model for analysis. our argument is that a usable index of a non-boolean retrieval model should meet the following three conditions: 1) the document content in the index should have a one-to-one corre-spondence with the id to ensure the stability of retrieval results; 2) the key information of the documents should be stored in the index as completely as possible to avoid the loss of information related to the query and thus affecting the retrieval results; 3) the model should be able to output documents in decreasing order of their relevance to the query. these three abilities are summarized as exclusivity, completeness, and relevance ordering. our analytical experiments indicate that the currently available dsi models do not fully meet the requirements of a general end-to-end indexingretrieval model, which limits their conditions of use and significantly reduces their effectiveness, especially when compared to the state-of-the-art dense retrieval models, which, in contrast, is shown to better meet those requirements. to this end, we investigate whether dsi models can be better trained to improve retrieval abilities while maintaining their simple structure and low storage cost. specifically, we propose utilizing a dense retrieval approach to provide effective supervision signals for training dsi models. to enhance exclusivity and completeness of dsi, we propose to improve the document representation to capture information from different granularities and filter key information using the document representation encoded by the dense retrieval model. to improve the ability to discriminate the relevance degree of different documents of dsi, we propose a new distillation-based training approach. by explicitly modeling the connections between documents, the model is able to reduce the randomness of the output results and improve retrieval performance, especially on datasets with deep pool annotations. major contributions of this paper are tri-fold. 1) an empirical analysis of basic ir abilities indicates the potential weaknesses of existing dsi approaches. 2) based on the analysis above, we propose a multi-task distillation approach to improve the effectiveness of dsi by learning from dense retrieval while keeping its advantages. 3) further evaluation shows that our approach substantially improves retrieval effectiveness of dsi-qg. ## empirical analysis in this section, we conduct an empirical analysis to examine to what extent the dsi framework satisfies the basic requirements of a functioning ir model. specifically, we summarize exclusivity, completeness, and relevance ordering, as three essential abil-ities required for an ir framework, as defined below. while acknowledging that there are certain retrieval problems that do not require an ordered list, it is crucial to emphasize that our research specifically focuses on ranked retrieval, which inherently involves non-boolean ranking. the notions used throughout this paper are listed in ## related works the field of information retrieval has recently seen a surge in interest in generative retrieval models. examples of this approach include the doct5query @xcite , which trains t5 models to generate document-related queries and adds them to the original documents for index construction, and seal @xcite , which generates text fragments that the target document may contain and uses fm-index @xcite for retrieval. genre @xcite utilizes bart @xcite for entity name generation in entity retrieval tasks. furthermore, @xcite proposed the differentiable search index (dsi), which explores various strategies for mapping string queries to document ids based on t5 @xcite . this idea has been further developed in models such as dsi-qg @xcite , nci @xcite , and ultron @xcite , which have all effectively improved the retrieval performance of generative retrieval models through the use of pseudo queries. more specifically, dsi-qg @xcite adapted the model for multilingual retrieval tasks, ultron @xcite attempted to incorporate hyperlink information into document ids, and nci @xcite employed richer data, more fine-grained semantic id mapping, and a novel decoder structure to obtain the best retrieval results. differentiable indexing models have also been applied to a wide range of tasks such as knowledge-intensive language tasks @xcite , long sequence retireval @xcite and qa tasks @xcite . additionally, efforts have been made to improve the memory capability @xcite and decoding capability @xcite of these models. this paper aims to gain a deeper understanding of differentiable indexing models as a retrieval method and proposes a multi-task distillation approach to improve their performance. ## conclusions we summarized three essential abilities of a functional non-boolean ir framework. through an empirical analysis of these abilities, we identified potential weaknesses in existing dsi approaches. to address these weaknesses, we propose a multitask distillation approach to enhance the effectiveness of dsi by learning from dense retrieval while preserving the advantages of dsi, such as minimal storage cost and end-to-end retrievability. our evaluation results indicate that our proposed approach improves the three abilities of dsi and, as a result, its retrieval effectiveness, particularly on data with more comprehensive human annotations. there are several directions to explore. firstly, the three capabilities identified in our empirical analysis, although currently applied to dsi, can be further extended to analyze statistical retrieval methods as well. additionally, investigating the trade-off between these three capabilities would be an interesting avenue for future research. performing more comprehensive experiments to ascertain the effectiveness of retrieval models that rely on expanded queries for downstream tasks would yield valuable insights.