Re3val: Reinforced and Reranked Generative Retrieval (2024)

Abstract: Generative retrieval models encode pointers to information in a corpus as an index within the model’s parameters. These models serve as part of a larger pipeline, where retrieved information conditions generation for knowledge-intensive NLP tasks. However, we identify two limitations: the generative retrieval does not account for contextual information. Secondly, the retrieval can’t be tuned for the downstream readers as decoding the page title is a non-differentiable operation. This paper introduces Re3val, trained with generative reranking and reinforcement learning using limited data. Re3val leverages context acquired via Dense Passage Retrieval to rerank the retrieved page titles and utilizes REINFORCE to maximize rewards generated by constrained decoding. Additionally, we generate questions from our pre-training dataset to mitigate epistemic uncertainty and bridge the domain gap between the pre-training and fine-tuning datasets. Subsequently, we extract and rerank contexts from the KILT database using the rerank page titles. Upon grounding the top five reranked contexts, Re3val demonstrates the Top 1 KILT scores compared to all other generative retrieval models across five KILT datasets.

## introduction the primary objective of retrieval models is to enhance the accuracy of answers by selecting the most relevant documents retrieved for a given query, ensuring models have sufficient information to help the downstream reasoning process. for instance, drqa @xcite introduces a "retrieve and read" pipeline using tf-idf to return documents for a question answering model to achieve this goal. more recently, nlp researchers have studied neural retrieval models like dense passage retrieval (dpr) (karpukhin et al., 2020) † work performed while at kaist ai. with a seq2seq model to build retrieval augmented language models. rather than using inner-product-based retrieval, generative retrieval models such as genre @xcite and corpusbrain @xcite generate page titles through constrained decoding, attaining higher r-precision and recall compared to dpr. in our work, we further evaluate how additional contextual information can benefit the generative retrieval models through reranking and how reinforcement learning can enhance relevance through reward signals. we introduce re3val: reinforced and reranked generative retrieval, a novel framework specifically designed to address the challenges in neural information retrieval. our approach utilizes 500k pre-training data and 48k task-specific data for training. despite the reduced data used in distant supervision, re3val achieves exceptional performance. our contributions are described as below: in summary, re3val uses dpr contexts for reranking page titles, leading to improved r-precision. re3val enhances performance by integrating generated questions in pre-training and utilizing reinforce during distant supervision. moreover, re3val achieves more accurate answers by reading reranked contexts retrieved with the reranked page titles. these advancements enable re3val to achieve state-of-the-art performance while also offering cost savings by reducing training time and minimizing the need for extensive data labeling. ## methodology the primary contribution of re3val is its capability to generatively rerank page titles by incorporating contextual information and to apply reinforce during distant supervision of a generative retrieval. additionally, re3val utilizes question generation for pre-training. furthermore, re3val pioneers the reading of contexts retrieved using page titles obtained through a generative retrieval approach. the following elucidates the function of each component in figure 2 with respect to its task.