Breaking Boundaries in Retrieval Systems: Unsupervised Domain Adaptation with Denoise-Finetuning (2023)

Abstract: Dense retrieval models have exhibited remarkable effectiveness, but they rely on abundant labeled data and face challenges when applied to different domains. Previous domain adaptation methods have employed generative models to generate pseudo queries, creating pseudo datasets to enhance the performance of dense retrieval models. However, these approaches typically use unadapted rerank models, leading to potentially imprecise labels. In this paper, we demonstrate the significance of adapting the rerank model to the target domain prior to utilizing it for label generation. This adaptation process enables us to obtain more accurate labels, thereby improving the overall performance of the dense retrieval model. Additionally, by combining the adapted retrieval model with the adapted rerank model, we achieve significantly better domain adaptation results across three retrieval datasets. We release our code for future research.

## introduction the goal of information retrieval (ir) is to enable users to input a query and retrieve relevant passages or documents from the retrieval system. a standard ir system @xcite @xcite typically comprises two main stages (refer to fig. 1 ): recent advancements in contextualized word embeddings @xcite @xcite have established dense retrieval figure 1 : two-stage information retrieval involves using a retrieval model to select a small subset of passages, which are then reranked using a rerank model. @xcite as the mainstream approach in information retrieval (ir). this approach effectively addresses the issue of missing words encountered in lexical methods like bm25 @xcite or tf-idf @xcite and has powerful retrieval ability. however, the introduction of the beir benchmark @xcite has highlighted the limitations of dense models. surprisingly, traditional lexical-based models like bm25 have exhibited superior performance compared to dense retrieval models in out-of-domain scenarios. additionally, training dense retrieval models from scratch necessitates a significant amount of domain-specific training data, posing challenges in collecting largescale data for each specific domain. as a result, there is a growing demand for domain adaptation methods that can enhance model performance without relying on labeled data. previous unsupervised domain adaptation approaches, such as qgen @xcite and gpl @xcite , have employed generative models to generate pseudo queries and augment the training data. qgen and gpl utilized a finetuned rerank model @xcite on the msmarco dataset @xcite to assign relevance scores to the generated queries and passages, creating a pseudo dataset for training the dense retrieval model. however, they did not adapt the rerank model to the target domain, resulting in potential label imprecision and failure to improve the two-stage retrieval score. in this paper, we present a novel technique for unsupervised domain adaptation (fig. 2 ). our approach focuses on simultaneously adapting the dense retrieval model and rerank model within a two-stage retrieval system. the process begins by generating pseudo queries for target domain passages, treating them as relative pairs. we employ sparse and dense negative miners to identify challenging negatives and create a pseudo dataset by combining queries with relevant and irrelevant passages. the rerank model is then fine-tuned using this dataset. since this dataset is not labeled by human, it is likely to contain noisy labels. therefore, we propose a denoise-finetuning approach that incorporates random batch warm-up and co-regularization learning. our experimental results validate the effectiveness of this approach in alleviating the impact of noisy labels. in our domain adaptation approach, we utilize models that have been fine-tuned on the extensive msmarco @xcite ir dataset, which serves as a strong foundation for adapting to various domains. remarkably, our approach outperforms previous domain adaptation methods in both one-stage and two-stage retrieval tasks, showcasing superior performance. in summary, our pipeline offers the following contributions: ## methodology our method, illustrated in fig. 2 , achieves domain adaptation for both the retrieval model and rerank model components through three steps: (1) constructing a pseudo training set, (2) denoisefinetuning the rerank model for domain adaptation, and (3) distilling the knowledge from the rerank model to the dense retrieval model. ## conclusion in conclusion, we have presented a novel approach for unsupervised domain adaptation in information retrieval. our method addresses the limitations of dense retrieval models and aims to enhance performance in out-of-domain scenarios. by simultaneously adapting the rerank model and dense retrieval model, we achieve significant improvements in two-stage retrieval performance. experiment results demonstrate that our domain adaptation approach outperforms previous methods in both one-stage and two-stage retrieval tasks. the fine-tuned rerank model effectively captures domain-specific information, which is then transferred to the dense retrieval model through knowledge distillation. overall, our work contributes to the field of unsupervised domain adaptation in information retrieval and provides a valuable framework for improving retrieval performance across diverse domains without the need for domain-specific labeled data.