GLEN : Generative Retrieval via Lexical Index Learning (2023)

Abstract: Generative retrieval shed light on a new paradigm of document retrieval, aiming to directly generate the identifier of a relevant document for a query. While it takes advantage of bypassing the construction of auxiliary index structures, existing studies face two significant challenges: (i) the discrepancy between the knowledge of pre-trained language models and identifiers and (ii) the gap between training and inference that poses difficulty in learning to rank. To overcome these challenges, we propose a novel generative retrieval method, namely Generative retrieval via LExical iNdex learning (GLEN). For training, GLEN effectively exploits a dynamic lexical identifier using a two-phase index learning strategy, enabling it to learn meaningful lexical identifiers and relevance signals between queries and documents. For inference, GLEN utilizes collision-free inference, using identifier weights to rank documents without additional overhead. Experimental results prove that GLEN achieves state-of-the-art or competitive performance against existing generative retrieval methods on various benchmark datasets, e.g., NQ320k, MS MARCO, and BEIR. The code is available at https://github.com/skleee/GLEN.

## introduction generative retrieval has emerged as an innovative approach to document retrieval @xcite . unlike conventional retrieval methods following the "index-retrieve-then-rank" pipeline, it unifies an entire search process. specifically, it directly generates the identifier of a relevant document for a given query. by formulating the entire search process as a sequence-to-sequence problem, it bypasses an auxiliary index structure and can be optimized through end-to-end learning. despite these benefits, generative retrieval faces major challenges in how to define and train docu- * equal contribution † corresponding author ment identifiers. as depicted in to this end, we introduce a new generative retrieval method, namely generative retrieval via lexical index learning (glen) using the dynamic lexical identifier in a right-bottom cell in the key novelty of glen is (i) to define lexical identifiers from documents using the knowledge of plms, (ii) to learn them from relevance between queries and documents, and (iii) to effectively rank documents with the same identifier for inference. ## proposed method in this section, we formulate the generative document retrieval task (section 3.1) and present glen (section 3.2), as depicted in figure 1 . to tackle the challenges of identifier design and training strategy, glen adopts a two-phase lexical index learning (section 3.3). for inference, we devise a collisionfree inference using identifier logits (section 3.4). while maintaining its simplicity, glen handles identifier collisions where semantically similar documents share the same lexical identifier. ## conclusion in this paper, we proposed a novel lexical index learning method, namely generative retrieval via lexical index learning (glen). to effectively tackle the critical challenges of generative retrieval, we adopt a dynamic lexical identifier learning framework that can mitigate (i) the discrepancy between the knowledge of pre-trained language models and identifiers and (ii) the discrepancy between training and inference. glen successfully enjoys the benefits of a dynamic lexical document identifier via a delicately devised two-phase index learning scheme and collision-free inference. to our knowledge, it is the first work introducing a dynamic lexical identifier for generative retrieval. experimental results demonstrate that glen achieves state-of-the-art or competitive performance among generative retrieval methods.