Nonparametric Decoding for Generative Retrieval (2023)

Abstract: The generative retrieval model depends solely on the information encoded in its model parameters without external memory, its information capacity is limited and fixed. To overcome the limitation, we propose Nonparametric Decoding (Np Decoding) which can be applied to existing generative retrieval models. Np Decoding uses nonparametric contextualized vocab embeddings (external memory) rather than vanilla vocab embeddings as decoder vocab embeddings. By leveraging the contextualized vocab embeddings, the generative retrieval model is able to utilize both the parametric and nonparametric space. Evaluation over 9 datasets (8 single-hop and 1 multi-hop) in the document retrieval task shows that applying Np Decoding to generative retrieval models significantly improves the performance. We also show that Np Decoding is data- and parameter-efficient, and shows high performance in the zero-shot setting.

## introduction text retrieval is often formulated as finding the most relevant items from a large corpus given an input query. the bi-encoder approach of using an encoder to map the documents and the query to a common vector space and performing a nearest neighbor search has been a common practice in text retrieval tasks @xcite @xcite . despite its high performance and popularity, it has an embedding space bottleneck @xcite ; limited expressiveness due to fixed-size embeddings and misses the fine-grained interaction between embeddings as they interact in l2 or inner product space. moreover, the bi-encoder approach requires large storage space to save all document embeddings. a recently-proposed alternative to the bi-encoder approach is using a generative retrieval model @xcite @xcite @xcite @xcite . it is an autoregressive model that retrieves the most relevant sequence by generating the target sequence (e.g., title, passage, document id) token-by-token. it overcomes the embedding space bottleneck by interacting in the parametric space. also, it is storage efficient by not having any external memory. however, the information capacity of such fully parametric models tends to be bounded by their sizes as it has to encode all information in its parameters @xcite . to this end, we propose nonparametric decoding (np decoding), a decoding method for generative retrieval models. it uses nonparametric contextualized vocab embeddings rather than vanilla vocab embeddings as decoder vocab embeddings. the contextualized vocab embeddings are output embeddings of an encoder that constructs a nonparametric dense vector space and are frozen during the training step whereas the vanilla vocab embeddings are trainable model vocab embeddings that construct a parametric space of the model. therefore, by using np decoding, the generative retrieval model does not have to rely solely on its own parameters but can utilize the surrounding information encoded in the contextualized vocab embeddings (external memory). note that while it utilizes the dense vector space as in the bi-encoder approach, unlike the approach, it does not have embedding space bottleneck as it is a variant of the generative retrieval model, and saves storage space by storing only clustering centroid embeddings (section 3.5). as shown in figure 1 , any generative retrieval model can incorporate np decoding by replacing the decoder vocab embeddings from the vanilla embedding matrix to contextualized embedding vanilla embedding space contextualized embedding space . cape . town . cape . south . africa . of . climate . cape . town . climate . of . africa . south : vanilla : w/ np decoding "during which season does cape town receive rainfall" softmax climate of south africa vanilla embedding matrix cape cape … climate town south africa cape town: cape town is one of south africa's three matrix (ce) for both the training and the inference steps. by the replacement, np decoding has two key benefits over vanilla decoding. first, the generative retrieval model can utilize not only its parametric space but also its nonparametric space. the nonparametric space is constructed with decoder vocab embeddings of np decoding (ce), nonparametric and context-aware embeddings that capture surrounding information. second, ce allows a token to have multiple token embeddings, unlike vanilla vocab embeddings where a token has a unique embedding. therefore, the decoder vocab embedding space of ce becomes more expressive and fine-grained (right side of figure 1 ). since having a well-constructed ce is important for achieving high performance, we propose three different encoders (ce encoder) used to output contextualized vocab embeddings added to ce (section 3). we demonstrate that ce encoder with contrastive learning results in a significant increase in performance. the main contributions of our paper are as follows: generative retrieval generative retrieval models retrieve relevant items by generating sub/either the identifiers or entire sequences of the items. genre @xcite retrieves a document by generating the titles with a constrained beam search. dsi @xcite assigns a unique id to each item in the corpus and retrieves the item by generating the id of the most relevant document. seal @xcite retrieves any span from any position in the corpus by using fm-index. gmr @xcite retrieves the most relevant item by generating the whole sequence. though high performance, as generative retrieval models solely rely on the information stored in their parameter, the information capacity is limited and fixed. to overcome the limitation, we propose nonparametric decoding (np decoding) for generative retrieval models. by replacing the decoder vocab embeddings with nonparametric contextualized vocab embeddings, the model is able to utilize not only the parametric space but also the nonparametric space of contextualized embeddings. memory augmented models knn-lm (khandelwal et al., 2020), trime @xcite , rag @xcite , and retro @xcite are memory augmented models which use both the parametric space of the model and the non-parametric space of the external memory. knn-lm improves the lm performance by generating the next token through interpolation between the nearest neighbor distribution (distance in the contextualized embedding space) and the model vocab distribution only during the inference step. trime expands the work to use the objective also during the training step. rag and retro first retrieve relevant texts with the retriever from the external memory and generate the output based on the retrieved texts. moreover, concurrent work npm @xcite proposes a nonparametric masked language model which operates over the nonparametric distribution of the external memory. generative retrieval models with nonparametric decoding also utilize the external memory, but rather than considering it as an external source, it is incorporated with the model by utilizing the external memory as decoder vocab embeddings. ## nonparametric decoding generative retrieval is the task of retrieving the most relevant retrieval target (e.g., title, passage, document identifier) by generating the target tokenby-token when given an input query. the training objective of the generative retrieval model is to maximize to address the limitation, we propose a new decoding method called nonparametric decoding (np decoding) for generative retrieval. to incorporate np decoding on the existing generative retrieval model, the only amendment is to use the frozen contextualized vocab embedding (external memory) rather than the vanilla vocab embedding as the decoder vocab embedding during each generation step (figure 1 ). the embeddings are the output embeddings of an encoder when given a target sequence as input. note that existing generative retrieval models such as genre and dsi utilize the pre-trained language model architecture as-is: vanilla vocab embedding as the decoder vocab embedding. in section 3.1, we show the key benefits of using np decoding over vanilla decoding. for section 3.2 to section 3.4, we show the details of base np decoding (base), and two variants (async, contra). in section 3.5, we describe how we reduce the number of contextualized token embeddings. ## experimental setup in section 4.1 and section 4.2, we describe the baselines and the datasets we used for experiments. in section 4.3, we show how we construct ce depending on which generative retriever we combined with. we experiment over 9 datasets. results show that by simply replacing the decoding strategy, generative retrieval shows significantly higher performance. see appendix b for more details about setups. ## experimental results in this section, we demonstrate the benefits of using np decoding in generative retrieval models by comparing the performance of existing generative retrieval models (dsi, genre*) with and without the method in document retrieval tasks. 9 genre* is genre trained with t5-large. we used t5 as a base model for a fair comparison with other base models. see appendix c.2 for a performance comparison between genre* (t5-large) and genre (bart-large). 10 we use the first five paragraphs of the document to the maximum of 512 tokens as the document content due to limited input length. 11 as the document id of official dsi is not released, we assign the document id with arbitrary unique integers (naively structured identifiers, dsinaive). dsinaive with our document id shows 12.5 and 22.4 for hits@1 and hits@10. ## conclusion in this paper, we propose nonparametric decoding (np decoding), a new decoding method that can be applied to canonical generative retrieval models by simply replacing the decoder vocab embeddings from vanilla vocab embeddings to nonparametric contextualized vocab embeddings (output embeddings of an encoder). this way, the generative retrieval does not rely solely on the information encoded in its own model parameters but can also utilize the information encoded in the contextualized embeddings. using np decoding in generative retrieval significantly improves the performance, achieves higher data and parameter efficiency, and shows more robustness in the zeroshot setting. in future work, we plan to apply np decoding to various other tasks beyond task retrieval.