When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using G rad N orm IR (2025)

Abstract: Dense retrievers encode texts into embeddings to efficiently retrieve relevant documents from large databases in response to user queries. However, real-world corpora continually evolve, leading to a shift from the original training distribution of the retriever. Without timely updates or retraining, indexing newly emerging documents can degrade retrieval performance for future queries. Thus, identifying when a dense retriever requires an update is critical for maintaining robust retrieval systems. In this paper, we propose a novel task of predicting whether a corpus is out-of-distribution (OOD) relative to a dense retriever before indexing. Addressing this task allows us to proactively manage retriever updates, preventing potential retrieval failures. We introduce GradNormIR, an unsupervised approach that leverages gradient norms to detect OOD corpora effectively. Experiments on the BEIR benchmark demonstrate that GradNormIR enables timely updates of dense retrievers in evolving document collections, significantly enhancing retrieval robustness and efficiency.

## introduction with the exponential growth of digital content, information retrieval (ir) systems have become essential for delivering relevant information from massive document repositories @xcite . unlike traditional sparse retrieval methods @xcite that rely heavily on lexical overlap, dense retrievers @xcite utilize semantic embeddings to better understand query intent and retrieve documents with similar conceptual meanings, thus overcoming the constraints of exact term matching. consequently, dense retrievers have gained significant attention in tasks demanding high semantic precision, such as question answering and personalized search. during training, these dense retrievers q. what's a song from taylor's ninth album? q. tell me how the "willow" works! evolving corpora q. what's the name of taylor's tenth album? dense retriever corpus gradient norm gradnormir "willow" is a song by taylor swift from her ninth … midnights is the tenth studio album by taylor swift … google introduced new quantum chip "willow." it … "willow" is a song by taylor swift from her ninth … midnights is the tenth studio album by taylor swift … "willow" is a song by taylor swift from her ninth … dense retriever dense retriever dense retriever figure 1 : motivation. in evolving corpora, indexing documents that dense retrievers fail to generalize to can severely degrade retrieval performance. therefore, proactively detecting ood corpora prior to indexing, without relying on available queries, is essential for maintaining retrieval effectiveness. to address this, we propose gradnormir, an unsupervised method leveraging gradient norms to identify such ood corpora. in the real world, corpora evolve rapidly due to technological advancements, societal changes, and emerging trends. this continuous evolution poses a substantial challenge for dense retrievers @xcite , which often struggle to generalize effectively to unseen documents in zero-shot scenarios @xcite . the problem is particularly critical in retrieval-augmented generation (rag) systems @xcite , where retriever performance directly impacts downstream tasks @xcite @xcite . for instance, consider the scenario illustrated in figure . 1: when a new corpus about google's quantum computing chip, willow, is introduced, a dense retriever trained on existing content, such as the song willow by taylor swift, may erroneously retrieve irrelevant documents. a query like "tell me how the willow works!" could mistakenly return information about the song willow instead of the quantum chip. anticipating when a retriever might fail due to such distributional shifts is crucial. also, proactively identifying when to update the retriever ensures robustness and accuracy in dynamically evolving document streams. this challenge closely relates to the ood generalization problem. several approaches in information retrieval (ir) aim to improve a retriever's performance on queries or documents significantly different from those encountered during training @xcite @xcite @xcite @xcite . one prominent approach utilizes a mixture-of-experts framework, employing a gating mechanism to select the most suitable expert retriever for each test query @xcite . however, these methods depend heavily on predefined expert retrievers trained offline with explicit domain knowledge and established domain boundaries, making them less flexible for dynamically evolving corpora. determining the appropriate timing to introduce new experts for emerging content remains challenging. to address this challenge, we introduce a novel practical task of predicting ood corpora before indexing for a given dense retriever. identifying ood corpora in advance signals when retriever updates are needed. by detecting such corpora before indexing, we can proactively select a more suitable dense retriever or promptly update the current one, safeguarding inference-time performance. to achieve this, we propose gradnormir, an unsupervised method designed to detect ood documents within a corpus without relying on queries. inspired by successful applications in image classification, where gradient norms effectively detect ood images and estimate test-time accuracy without labeled data @xcite , gradnormir leverages gradient norms from the contrastive loss to evaluate a retriever's generalizability on a given corpus. specifically, we introduce novel sampling strategies to effectively assign positive and negative instances for computing the contrastive loss. we assess our method on the beir benchmark, which encompasses multiple diverse datasets across various domains. firstly, we demonstrate that gradnormir effectively detects ood documents likely to cause retrieval failures. subsequently, we show that gradnormir can select the most suitable retriever using only the corpus, without queries. finally, we simulate evolving corpora using beir by sequentially introducing datasets following @xcite , and demonstrate how gradnormir enables efficient retriever updates while maintaining performance. our experiments validate both the importance of ood detection for retrieval systems and gradnormir's efficacy in adapting to evolving corpora. in summary, our contributions are as follows: ## related work information retrieval. recent advancements in text embeddings have significantly transformed the field of ir, particularly with the emergence of dense retrievers. the success of these models has primarily been driven by the availability of largescale training datasets, including nq @xcite , ms-marco @xcite , hot-potqa @xcite , and nli @xcite . a notable example is dpr @xcite , which employs a dual-encoder architecture for open-domain question-answering, independently embedding queries and passages. unsupervised approaches have also garnered attention for their ability to enhance the generalization of dense retrievers. contriever @xcite enlarges pre-training data using unsupervised data augmentation techniques for contrastive learning. similarly, e5 @xcite leverages weak supervision to create a large-scale dataset, ccpairs, filtered by consistency criteria. recently, hybrid methods like bge-m3 @xcite have integrated dense, sparse, and multi-vector retrieval strategies via self-knowledge distillation, further advancing retrieval effectiveness. ood robustness. in ir, ood robustness refers to a model's capacity to maintain retrieval effectiveness when exposed to documents that deviate significantly from its training distribution. a prominent benchmark to evaluate this robustness is beir @xcite , which encompasses diverse retrieval tasks across multiple domains. using beir, @xcite demonstrated that dense retrievers often underperform on ood datasets compared to traditional lexical retrievers like bm25. in response, they proposed a hybrid model combining dense and sparse retrieval, achieving robust performance in zero-shot scenarios. similarly, @xcite showed that distribution shifts lead to a substantial decline in zero-shot accuracy of dense retrievers. various strategies have been proposed to improve ood performance on unseen documents. data augmentation has yielded promising results @xcite . architectural adaptations have also been explored; mixtureof-experts frameworks @xcite and multi-head rag models @xcite adjust retrieval strategies based on domain characteristics. moreover, @xcite investigated zero-shot retrieval methods for selecting the most suitable retriever, while @xcite proposed leveraging llm-generated pseudo-queries to rank dense retrievers. other studies @xcite have adopted continual learning methods to handle dynamic corpora without forgetting previously acquired knowledge. for instance, memorybased approaches @xcite maintain backward compatibility with existing document embeddings, and incremental indexing strategies @xcite dynamically update document indices in generative retrievers to accommodate new information. despite these advancements, few studies have explicitly examined how to identify ood documents from the perspective of dense retrievers. layer-wise score aggregation methods, such as those proposed by @xcite , combine anomaly scores from each encoder layer to provide more accurate detection. however, this approach is primarily tailored to text classification tasks, whereas our work explicitly addresses evaluating the generalizability of dense retriever models across evolving document corpora. ## approach we first review prior work leveraging gradient norms in image classification. we then introduce gradnormir, an unsupervised method to proactively detect ood documents and consequently identify ood corpora. ## experiments we conduct three sets of experiments to evaluate our approach. first, next, we verify that grad-normir's ood detection is effective for selecting the most suitable retriever. next, we make sure that gradnormir's ood detection is useful in selecting the most suitable retriever, even without any queries. finally, in evolving corpora, we demonstrate that gradnormir enables efficient continuous retriever updates by selectively retraining it only on the predicted ood corpus. we also present an ablation study for several hyperparameters in appendix c. ## conclusion we introduced the novel task of predicting an ood corpus for dense retrievers before indexing, addressing a critical challenge for maintaining robust retrieval performance in dynamically evolving corpora. to tackle this problem, we proposed grad-normir, an unsupervised method leveraging gradient norms of the contrastive loss to proactively detect ood documents. by employing novel sampling strategies, including document-to-document retrieval with carefully selected positive and hard negative samples, gradnormir can effectively predict whether a corpus might pose retrieval challenges even before query collection. our method enables timely selection of the most suitable retriever or updates to an existing retriever, thereby ensuring robust performance in evolving document environments. an interesting direction for future research would involve online ood document detection, where documents arrive continuously rather than as a complete corpus. such an extension could enhance the practical applicability of retrievers in real-time retrieval scenarios.