Distillation Enhanced Generative Retrieval (2024)

Abstract: Generative retrieval is a promising new paradigm in text retrieval that generates identifier strings of relevant passages as the retrieval target. This paradigm leverages powerful generative language models, distinct from traditional sparse or dense retrieval methods. In this work, we identify a viable direction to further enhance generative retrieval via distillation and propose a feasible framework, named DGR. DGR utilizes sophisticated ranking models, such as the cross-encoder, in a teacher role to supply a passage rank list, which captures the varying relevance degrees of passages instead of binary hard labels; subsequently, DGR employs a specially designed distilled RankNet loss to optimize the generative retrieval model, considering the passage rank order provided by the teacher model as labels. This framework only requires an additional distillation step to enhance current generative retrieval systems and does not add any burden to the inference stage. We conduct experiments on four public datasets, and the results indicate that DGR achieves state-of-the-art performance among the generative retrieval methods. Additionally, DGR demonstrates exceptional robustness and generalizability with various teacher models and distillation losses.

## introduction text retrieval is a crucial task in information retrieval and has a significant impact on various language systems, including search ranking @xcite , open-domain question answering @xcite , and retrieval augmented generation (rag) @xcite . in recent years, dense retrieval @xcite has been the dominant approach for text retrieval based on the advancements in encoderbased language models, like bert (kenton and toutanova, 2019). ## method in this section, we begin by giving a preliminary of a standard generative retrieval system. following this, we outline our distillation framework as it applies to the generative retrieval system. ## conclusion and future work in this study, we proposed the distillation enhanced generative retrieval (dgr) framework, which could significantly improve generative retrieval models by distilling knowledge from advanced rank models. importantly, dgr is highly adaptable and can effectively work with different teacher models and distillation losses. additionally, we introduced the distilled ranknet loss, which is tailored to generative retrieval and has been evident to be more effective than existing distillation losses. we conducted extensive experiments on four widely used benchmarks to verify the effectiveness and robustness of dgr. the results verify that distillation is a promising direction for improving generative retrieval systems. in the future, we aim to improve dgr from the following aspects. we have investigated the use of cross-encoder and dual-encoder as teacher models, and we will also explore the possibility of advanced generative retrieval models as teacher models for outdated generative retrieval models. additionally, dgr has only begun to validate the feasibility of distillation in generative retrieval, leaving ample room for further research. further studies, including more sampling strategies and progressive training methods, are worth exploring.