DUQG en: Effective Unsupervised Domain Adaptation of Neural Rankers by Diversifying Synthetic Query Generation (2024)

Abstract: State-of-the-art neural rankers pre-trained on large task-specific training data such as MS-MARCO, have been shown to exhibit strong performance on various ranking tasks without domain adaptation, also called zero-shot. However, zero-shot neural ranking may be sub-optimal, as it does not take advantage of the target domain information. Unfortunately, acquiring sufficiently large and high quality target training data to improve a modern neural ranker can be costly and time-consuming. To address this problem, we propose a new approach to unsupervised domain adaptation for ranking, DUQGen, which addresses a critical gap in prior literature, namely how to automatically generate both effective and diverse synthetic training data to fine tune a modern neural ranker for a new domain. Specifically, DUQGen produces a more effective representation of the target domain by identifying clusters of similar documents; and generates a more diverse training dataset by probabilistic sampling over the resulting document clusters. Our extensive experiments, over the standard BEIR collection, demonstrate that DUQGen consistently outperforms all zero-shot baselines and substantially outperforms the SOTA baselines on 16 out of 18 datasets, for an average of 4% relative improvement across all datasets. We complement our results with a thorough analysis for more in-depth understanding of the proposed method’s performance and to identify promising areas for further improvements.

## introduction large language models (llms) have enabled new state-of-the-art performance in neural ranking @xcite @xcite ). an effective approach has been to train the llms on a large-scale general ranking task such as ms-marco passage or document ranking @xcite or wikipedia retrieval @xcite , to learn task-specific features, which are often shared across other domains and datasets. the resulting rankers can then be used, without any adaptation (or in a zero-shot way) for a wide range of ranking tasks. for example, the beir @xcite benchmark had demonstrated sota or near-sota performance of several zero-shot neural rankers on a diverse set of retrieval tasks. however, when switching to specialized domains such as finance or scientific documents, zeroshot ranking performance should benefit from additional information for the target domain. for training modern neural rankers, acquiring sufficiently large and high-quality target training data of querydocument pairs, to improve a modern neural ranker, can be costly and time-consuming. hence, there has been significant interest in various approaches to domain adaptation for neural rankers, with varying degrees of supervision, including unsupervised approaches using synthetically generated queries, documents, or both query-document pairs @xcite @xcite @xcite . unfortunately, most of the previously reported results did not exceed the ranking performance compared to the current sota zero-shot models, as evaluated on the beir benchmark @xcite . in other words, to the best of our knowledge, no previously reported unsupervised ranking adaptation method demonstrated consistent improvements over large neural sota zero-shot rankers. in this work, we investigate whether it possible to improve the ranking performance of a pretrained sota neural ranker for a given target domain, through unsupervised domain adaptation (uda). to address this question, we first identify a critical requirement for the synthetic training data to be effective for ranking adaptation: that the generated training data should be both representative of the target domain, and sufficiently diverse to force changes to the ranking model at the appropriate representation level, without causing over-fitting or catastrophic forgetting (i.e., degrading performance on the original ranking tasks). specifically, we propose a new method duq-gen, which stands for diversified unsupervised query generation. duqgenintroduces a general approach for ranking domain adaptation, which focuses on selecting representative and diverse set of documents and query pairs for training a neural ranker. duqgen only requires access to a (part of) target document collection to be searched, and can improve any pre-trained neural ranker. duqgen is illustrated in figure 1 , and introduces the following innovations compared to previous unsupervised ranking adaptation approaches: (1) representing the target document collection as document clusters; (2) diversifying the synthetic query generation by probabilistic sampling over the resulting document clusters; and (3) prompting a large llm for query generation with in-context examples to generate queries from the selected documents. as we show experimentally, these innovations are responsible for consistent improvements over the previous sota baselines for ranking adaptation on almost all beir benchmarks, as well as consistent improvements over zero-shot performance of sota neural rankers. in summary, our contributions include: ## related work in this section, we discuss the prior works that help to establish the problem and navigate the solution. ## experiments in this section, we provide details of our experimental setup to demonstrate the effectiveness of duq-gen. ## results and discussion in this section, we present our main experimental results and delve into the key observations. we first describe our primary findings, reported using ndcg@10 in ## analysis in this section, we report our analysis of duq-gen's performance, which includes examining the need for clustering, confirming the choice of the query generator, and validating the quality of the generated queries. ## conclusions we proposed a general unsupervised domain adaptation method duqgen, which can be used to fine-tune any ranking model for a given target domains. duqgen introduced significant innovations over the previously reported unsupervised domain adaptation methods. specifically, duqgen proposes representing the target domain collection with document clustering; an effective method to diversify the synthetically generated queries, and an effective prompting strategy for using an llm to generate more effective and representative synthetic training data. we experimentally demonstrated that duqgen is both scalable and effective, as it uses only a few thousands of synthetic training examples, while consistently improves over the sota zero-shot rankers, and significantly outperforms the sota methods for unsupervised domain adaptation methods in most cases. we complemented the strong empirical performance of duq-gen with an in-depth analysis of the components to quantify their contributions. together, the presented techniques and experimental results significantly advance neural ranking adaptation, establish a new state-of-the-art in neural ranking, and suggest promising directions for future improvements.