Exploring the Practicality of Generative Retrieval on Dynamic Corpora (2024)

Abstract: Benchmarking the performance of information retrieval (IR) is mostly conducted with a fixed set of documents (static corpora). However, in realistic scenarios, this is rarely the case and the documents to be retrieved are constantly updated and added. In this paper, we focus on Generative Retrievals (GR), which apply autoregressive language models to IR problems, and explore their adaptability and robustness in dynamic scenarios. We also conduct an extensive evaluation of computational and memory efficiency, crucial factors for real-world deployment of IR systems handling vast and ever-changing document collections. Our results on the StreamingQA benchmark demonstrate that GR is more adaptable to evolving knowledge (4–11%), robust in learning knowledge with temporal information, and efficient in terms of inference FLOPs (x2), indexing time (x6), and storage footprint (x4) compared to Dual Encoders (DE), which are commonly used in retrieval systems. Our paper highlights the potential of GR for future use in practical IR systems within dynamic environments.

## introduction transformer-based information retrieval (ir) models play a vital role in advancing the field of semantic document search for informationseeking queries. notably, generative retrieval (gr) @xcite @xcite @xcite @xcite @xcite has recently gained a significant amount of recognition from the research community for its simplicity and high performance. however, dual encoder (de) (gillick et al., 2018; karpukhin et al., 2020a; ni et al., 2021; gao et al., * co-first authors. to this end, we establish dynamic information retrieval (dynamicir), a setup designed to simulate realistic scenarios for corpus updates in ir. this dynamicir setup includes two distinct update strategies, (1) indexing-based updates: updating only the index without any further pretraining or finetuning and (2) training-based updates: continually pretraining the parameters on new corpora in addition to updating the index (see figure 1 ). within this experimental setup, we evaluate the adaptability of recent state-ofthe-art retrieval models: seal @xcite , minder @xcite , and ltrgr @xcite for gr, and spider @xcite , contriever @xcite , and dpr @xcite for de. furthermore, we perform extensive comparison for the efficiency of each method, considering factors such as floating-point operations (flops) @xcite required for the inference, indexing time, inference latency, and storage footprint. the findings of our study underscore the strength of gr compared to de across three key aspects: adaptability, robustness, and efficiency. (1) gr exhibits superior adaptability to evolving corpora (section 5). gr outperforms de, showcasing 4 -11% greater adaptability in both indexing-based and training-based updates. notably, gr not only acquires new knowledge more effectively but also shows no sign of forgetting; rather, training with new corpora appears to enhance its existing knowledge. (2) gr is more robust without inducing undesired bias from data characteristics (section 5). de reveals a bias towards lexical overlap of timestamps inserted into queries and documents, showing significant degradation (52.23% ñ 17.40%) when the timestamps are removed. whereas, gr shows robust retrieval performance over temporal data. ## results and analysis in this section, we showcase the adaptability and temporal robustness of gr and de and provide an analysis on the effective continual pretraining approach and utilizing r 1 new during the trainingbased updates. we also discuss the performance of bm25 in dynamic environments. gr has greater adaptability in both update scenarios. we define adaptability as the ability of retrieval models to maintain the performance after the updates. to evaluate the adaptability, we examine the performance on q total in each update scenario and compare it with that on q initial in staticir (see first, in indexing-based updates, gr exhibits 4% greater adaptability to new corpora compared to de. specifically, when we look from q initial of staticir to q total , gr maintains average performance, while de demonstrates a 4% degradation on average. second, in training-based updates, gr efficiency is evaluated using 4 metrics on the right side. for inference latency, t online indicates the time required for query embedding and search, and t offline represents the time for loading the indexed corpus. we highlight the best scores in bold for each setup. additionally, the zero-shot performance for all models is provided in appendix 7. for de, we extract the update score from q w/o bias total instead of q total . because de exhibits a significant inherent bias towards the lexical overlap of timestamps when evaluating q new . we delve deeper into this phenomenon below. gr better acquires new corpora even without parameter updates. we assess the ability to acquire new knowledge through q new in both update scenarios. for de, we consider the performance of q w/o bias new instead of q new , reflecting the impact of bias as described above. in indexing-based updates, ## computation & memory efficiency in this section, we provide the results of computational and memory efficiency. we use an 80g a100 gpu. ## conclusion in this work, we explore the practicality of gr in terms of adaptability to new corpora, temporal robustness, and inference efficiency, compared to de. by establishing a dynamicir setup, we showcase how retrieval models perform in dynamic environments where knowledge evolves over time, with and without parameter updates. our findings indicate that gr better adapts to changing knowledge and less forgets by leveraging the power of language models. in terms of temporal robustness, gr retrieves information effectively regardless of data characteristics, while de struggles with temporal information. additionally, gr has fewer inference flops, reduced indexing time, and lower memory requirements. by comprehensively exploring gr across these three aspects, we underscore its potential as an ir system.