Enhancing Generative Retrieval with Reinforcement Learning from Relevance Feedback (2023)

Abstract: The recent advent of end-to-end generative retrieval marks a significant shift in document retrieval methods, leveraging differentiable search indexes to directly produce relevant document identifiers (docids) in response to a specific query. Nevertheless, this approach faces two fundamental challenges: (i) a discrepancy between the token-level probabilistic optimization and the broader document-level relevance estimation; (ii) an overemphasis on top-1 results at the expense of overall ranking quality. To tackle these challenges, we propose a generative retrieval model with reinforcement learning from relevance feedback, which aims to align token-level docid generation with document-level relevance estimation. The training process incorporates three stages: supervised fine-tuning, relevance reward model training, and reinforced learning-to-rank from relevance feedback. To train a high-quality reward model, we define “relevance” under three progressive scenarios, which collectively offer a comprehensive evaluation of the document relevance. Experiments conducted on two benchmark datasets demonstrate the effectiveness of our proposed approach.

## introduction information retrieval systems have undergone progressive advancements over the past few decades @xcite . historically, the retrieval of candidate documents has been heavily reliant on the use of an inverted index @xcite , a conventional method that employs term-based parsing. with the emergency of neural networks, dense retrieval @xcite is proposed to encode both query and document into dense vectors, so as to measure the similarity in semantic space. recently, there has been a growing focus on investigating end-to-end generative retrieval models (tay et al., 2022; bevilacqua et al., 2022; wang * corresponding author @xcite . these models supersede the conventional explicit index by implementing a large-scale model known as the differentiable search index @xcite . this innovative approach facilitates end-to-end document retrieval using a seq-to-seq generative model, such as t5 @xcite . in this framework, the model directly generates the corresponding docid for a given query. despite the notable progress made in generative retrieval models, they still face two major challenges that limit their ability to retrieve relevant documents. first, these models primarily rely on the auto-regressive loss to maximize the token-level generation probability of the ground-truth docid. however, such an optimization goal is not aligned with the primary objective of ranking tasks, which is to estimate document-level relevance. second, existing approaches assume a deterministic distribution in which all probability mass is allocated to the "golden" docid. such an assumption can lead the model to overemphasize the top-1 generation results, while ignoring the quality of the overall ranking list. in light of these challenges, establishing alignment between generation probabilities and document relevance is a vital necessity in adapting generative models to ranking tasks. in this paper, we present a generative retrieval model with reinforcement learning from relevance feedback (genrrl). our objective is to align the generative retrieval model, trained specifically on the task of docid generation, with the model responsible for assessing document relevance. the training process comprises three pivotal steps: (1) supervised fine-tuning. this initial step involves training the model using labeled data to improve its performance in generating docid. (2) relevance reward model training. the subsequent step focuses on training a relevance reward model, specifically designed to measure the degree of relevance between the generated docids and their correspond-ing queries. we expect the reward model to assign higher scores to docids that are more relevant to the queries. (3) reinforced learning-to-rank (l2r) from relevance feedback. the final step involves using the trained relevance reward model to guide the generative retrieval system towards generating more relevant docids. to align with the ranking task requirements, we incorporate the idea of l2r into the reinforcement learning procedure. this enables the provision of not only pointwise reward but also pairwise and listwise relevance feedback, thereby enhancing the overall ranking performance. in order to train a high-quality relevance reward model, it is crucial to define "relevance" accurately within the context of document retrieval tasks. in this regard, we focus on three key scenarios that robustly reflect the relationship between queries and documents. firstly, term-based overlap concerns the presence of identical or similar words within both the query and the document. techniques like tf-idf can effectively gauge this overlap. secondly, semantic similarity extends beyond mere term overlap, delving into the shared meanings or concepts within the query and the document. dual encoder models are particularly useful in modeling the semantic similarity by encoding the query and the document into dense vectors. lastly, contextual dependency considers how effectively the document satisfies the information needs expressed in the query, extending beyond the boundaries of specific terms or concepts within the query. techniques for capturing contextual dependency could include language models, which have demonstrated proficiency in understanding the contextual relationships between two sentences. by considering these three progressive scenarios, we can comprehensively assess the relevance between queries and documents, leading to a more effective reward model. to assess the effectiveness of our model, we perform comprehensive experiments on widely adopted ms marco and nq document retrieval datasets. experimental results demonstrate the efficacy of the proposed method, including the diverse relevance annotations and the reinforcement learning process. our in-depth analysis on different model sizes also reveals the scaling laws of our proposed method. our contributions are summarized as: (1) we employ reinforcement learning from relevance feedback to improve generative retrieval, aiming to align the docid generation process with document relevance assessment. (2) we outline a reinforced training process specifically tailored for the ranking task, which includes relevance reward model training and reinforced learning-to-rank. (3) we provide a classification of "relevance" within the context of document retrieval. by considering three progressive aspects, the trained reward model enables a thorough assessment of the relevance between queries and documents. ## related work traditionally, index-based document retrieval methods encompass both sparse and dense retrieval techniques. sparse retrieval methods @xcite @xcite , such as bm25 @xcite , utilize inverted indexes to assess term importance and calculate matching scores between queries and documents. dense retrieval methods have been employed to overcome the limitations of word mismatching @xcite . these methods convert queries and documents into dense vectors and employ anns @xcite for efficient vector-based search. the dual encoder approach incorporating pre-trained language models has shown improved performance in dense retrieval. techniques like hard negative sampling @xcite @xcite further enhance effectiveness. in recent times, generative retrieval has gained significant popularity in the field of information retrieval (ir) tasks. a model-based ir approach, where docids are embedded into the model, was first introduced in @xcite . since then, several other models @xcite @xcite incorporating a query augmentation module have been developed to enhance the performance. moreover, the model-based ir framework has been extended to knowledge-intensive language tasks, leading to notable improvements in performance @xcite @xcite . additionally, a series of studies by @xcite al. (2023 in this paper, we focus on aligning the optimizing objective of generative retrieval models with document-level relevance, so as to alleviate the bias against the objective of ranking tasks. figure 1 : empirical study on ranking results of generative retrieval and dense retrieval. ## preliminary in this section, we formulate the generative retrieval problem and investigate its limitations. ## methodology in this section, we focus on the training pipeline of genrrl, including supervised fine-tuning, relevance reward model training, and reinforced l2r from relevance feedback. ## experimental results and analysis in this section, we present the experimental findings of our proposed approach and perform an empirical analysis to provide comprehensive insights. ## conclusion in this paper, we address the challenges faced by generative retrieval models by proposing an rlbased approach called genrrl. our objective is to align the generative retrieval with the task of assessing document relevance, thereby improving ranking performance. we introduce a three-stage reinforcement learning framework, where the relevance reward model captures different dimensions of relevance by considering term-based overlap, semantic similarity, and contextual dependency. through extensive experiments, we demonstrate the effectiveness of our proposed method in improving generative retrieval performance.