D iffusion R et: Diffusion-Enhanced Generative Retriever using Constrained Decoding (2023)

Abstract: Generative retrieval, which maps from a query to its relevant document identifiers (docids), has recently emerged as a new information retrieval (IR) paradigm, however, having suffered from 1) the \textit{lack of the intermediate reasoning step} , caused by the manner of merely using a query to perform the hierarchical classification, and 2) the \textit{pretrain-finetune discrepancy} , which comes from the use of the artificial symbols of docids. To address these limitations, we propose the novel approach of using the document generation from a query as an intermediate step before the retrieval, thus presenting \underline{diffusion} -enhanced generative \underline{ret} rieval ( \textbf{DiffusionRet} ), which consists of two processing steps: 1) the \textit{diffusion-based document generation} , which employs the sequence-to-sequence diffusion model to produce a pseudo document sample from a query, being expected to semantically close to a relevant document; 2) \textit{N-gram-based generative retrieval} , which use another sequence-to-sequence model to generate n-grams that appear in the collection index for linking a generated sample to an original document. Experiment results on MS MARCO and Natural Questions dataset show that the proposed DiffusionRet significantly outperforms all the existing generative retrieval methods and leads to the state-of-the-art performances, even with much smaller number of parameters.

## introduction generative retrieval methods have emerged as powerful ir paradigms @xcite @xcite @xcite @xcite , by employing sequence-to-sequence models to map a query to relevant docids, showing promising results compared with dense retrieval based on dual encoders @xcite . in generative retrieval, because all the required information and knowledge for indexing and retrieval are maintained in parametric memory, retrieval is conducted efficiently at the inference stage by a decoder, without requiring other inverted index structures or tools for the nearest neighbor search, such as faiss @xcite . however, the classical methods of existing generative retrieval mostly produce only docids at inference time, much shorter than normal sentences, while no other content is generated @xcite @xcite . lacking an intermediate reasoning step, it is unclear what detailed reasoning steps proceed during decoding-based hierarchical classification foot_1 or docids generation, which is likely far from the improved approach based on the chain-of-thought @xcite . additionally, generative retrieval extensively uses artificial symbols of docids for target sequences. as noted in @xcite , however, these docid-like symbols are only seen during the finetuning but are unseen at the pretraining stage, thus causing the pretrain-finetune discrepancy, largely limiting the effect of transfer learning, which is arguably important when exploiting pretrained models. to address these limitations, we establish the document generation stage as a key intermediate reasoning step, that directly generates the "actual content" of a document relevant to a given query. once a document is generated, we use it as a new document-level query, generate n-grams restricted to those occurring in the collection, and use them to identify its original source document. to gen-erate such a pseudo query-relevant document, the diffusion model emerges as a noticeably promising alternative to the parameter-intensive large language models (llm) @xcite , surpassing some gpt variants even with a much smaller parameter scale, as in @xcite @xcite @xcite . pursuing the use of the reasoning chain of document-ngrams as an intermediate step, inspired by the aformentioned notable parameter efficiency of diffusion models, we propose a two-stage diffusion-enhanced generative retrieval (diffusionret) model, which consists of two processing steps, as follows: to the best of our knowledge, this is the first study where the diffusion model is deployed to generate a realistic document sample for the ir task. our contributions are summarized as follows: 1) we propose diffusionret, a two-stage generative retrieval model based on the reasoning chain of document-ngram, without requiring special tokens such as docids, 2) we apply a diffusion model for document generation as a textual version of the semantic inpainting, 3) the proposed diffusionret shows a state-of-the-art performance on the ms marco and natural questions (nq) datasets. ## method figure 1 presents the overall architecture of dif-fusionret used to form a document-ngram reasoning chain, consisting of 1) diffusion-based document generation and 2) ngram-based generative retrieval. formally, suppose that a query is given as x, @xmath0, • • • , d n } is a collection of documents where d i is i-th document, v is a vocabulary set of tokens, n is the maximum length of the sequence, and k is the fixed length of n-grams to be generated. diffusionret sequentially performs diffu(•), the diffusion-based document generation, and the n-gram generation, gen ngram (•) to compute the (3 ) and finally the rounding function to generate a pseudo-document q (i.e., eq. ( 4 )). ii) the n-gram generative retrieval applies gen ngram (q) to generate a set of n-grams q from q, under the restricted space of the fm-index of the collection and then computes the relevance score r q (d, q) as in eq ( 6) . ## conclusion in this paper, we proposed diffusionret, a twostage architecture consisting of diffusion-based document generation and n-gram-based generative retrieval. extensive experiments on the ms marco and nq datasets showed that diffusion-ret achieved state-of-the-art performances in terms of hits@1, outperforming most existing retrieval models. further analysis confirmed that the use of generated documents as a final query was the key component to bring about those significant improvements. in future work, we would like to invent a fullfledged diffusion model to jointly generate and retrieve without requiring a subsequent retrieval component. furthermore, we would also jointly perform a document generation and retrieval in a single transformer model by viewing document generation as the chain-of-thought @xcite . we would simplify refining n-gram-based generative retrieval such that the generation and retrieval components are less separated. comparing the full-fledged diffusion-based generate-then-retrieval model with popular llms would be interesting and inspire future research.