UDAPDR : Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers (2023)

Abstract: Many information retrieval tasks require large labeled datasets for fine-tuning. However, such datasets are often unavailable, and their utility for real-world applications can diminish quickly due to domain shifts. To address this challenge, we develop and motivate a method for using large language models (LLMs) to generate large numbers of synthetic queries cheaply. The method begins by generating a small number of synthetic queries using an expensive LLM. After that, a much less expensive one is used to create large numbers of synthetic queries, which are used to fine-tune a family of reranker models. These rerankers are then distilled into a single efficient retriever for use in the target domain. We show that this technique boosts zero-shot accuracy in long-tail domains and achieves substantially lower latency than standard reranking methods.

## introduction the advent of neural information retrieval (ir) has led to notable performance improvements on document and passage retrieval tasks @xcite @xcite as well as downstream knowledgeintensive nlp tasks such as open-domain questionanswering and fact verification @xcite @xcite . neural retrievers for these tasks often benefit from fine-tuning on large labeled datasets such as squad @xcite , natural questions (nq) @xcite , and kilt @xcite . however, ir models can experience significant drops in accuracy due to distribution shifts from the training to the target domain @xcite . for example, dense retrieval models trained on ms marco @xcite might not generalize well to queries about covid-19 scientific publications @xcite figure 1 : overview of udapdr. an expensive llm like gpt-3 is used to create an initial set of synthetic queries. these are incorporated into a set of prompts for a less expensive llm that can generate large numbers of synthetic queries cheaply. the queries stemming from each prompt are used to train separate rerankers, and these are distilled into a single colbertv2 retriever for use in the target domain. 2020), considering for instance that ms marco predates covid-19 and thus lacks related topics. recent work has sought to adapt ir models to new domains by using large language models (llms) to create synthetic target-domain datasets for fine-tuning retrievers @xcite @xcite . for example, using synthetic queries, @xcite and @xcite fine-tune the retriever itself and train a cross-encoder to serve as a passage reranker for improving retrieval accuracy. this significantly improves retriever performance in novel domains, but it comes at a high computational cost stemming from extensive use of llms. this has limited the applicability of these methods for researchers and practitioners, particularly in high-demand, user-facing settings. in this paper, we develop unsupervised domain adaptation via llm prompting and distillation of rerankers (udapdr), 1 an efficient strategy for using llms to facilitate unsupervised domain adaptation of neural retriever models. we show that udapdr leads to large gains in zero-shot settings on a diverse range of domains. the approach is outlined in figure 1 . we begin with a collection of passages from a target domain (no in-domain queries or labels are required) as well as a prompting strategy incorporating these passages with the goal of query generation. a powerful (and perhaps expensive) language model like gpt-3 is used to create a modest number of synthetic queries. these queries form the basis for corpus-adapted prompts that provide examples of passages with good and bad queries, with the goal of generating good queries for new target domain passages. these prompts are fed to a smaller (and presumably less expensive) lm that can generate a very large number of queries for fine-tuning neural rerankers. we train a separate reranker on the queries from each of these corpus-adapted prompts, and these rerankers are distilled into a single student colbertv2 retriever @xcite , which is evaluated on the target domain. by distilling from multiple passage rerankers instead of a single one, we improve the utility of colbertv2, preserving more retrieval accuracy gains while reducing latency at inference. our core contributions are as follows: ## methodology figure 1 outlines each stage of the udapdr strategy. for the target domain t , our approach requires access to in-domain passages (i.e., within the domain of t ). however, it does not require any in-domain queries or labels. the overall goal is to leverage our store of in-domain passages and llm prompting to inexpensively generate large numbers of synthetic queries for passages. these synthetic queries are used to train domain-specific reranking models that serve as teachers for a single retriever. the specific stages of this process are as follows: stage 1: we begin with a set of prompts that embed examples of passages paired with good and bad queries and that seek to have the model generate a novel good query for a new passage. we sample x in-domain passages from the target domain t , and we use the prompts to generate 5x synthetic queries. in our experiments, we test values of x such as 5, 10, 50, and 100. (in appendix a, we explore different strategies for selecting the x in-domain passages from the target domain.) in this stage, we use gpt-3 @xcite , specifically the text-davinci-002 model. the guiding idea is to use a very effective llm for the first stage, to seed the process with very high quality queries. we employ the five prompting strategies in figure 2 . two of our prompts are from @xcite , where they proved successful for generating synthetic queries in a few-shot setting. the remaining three use a zero-shot strategy and were inspired by prompts in @xcite . stage 2: the queries generated in stage 1 form the basis for prompts in which passages from the target domain t are paired with good and bad synthetic queries. the prompt seeks to lead the model to generate a good query for a new passage. our prompt template for this stage is given in figure 3 . we create y corpus-adapted prompts in this fashion, which vary according to the demonstrations they include. in our experiments, we test out several values for y , specifically, 1, 5, and 10. this programmatic creation of few-shot demonstrations for language models is inspired by the demonstrate stage of the dsp framework @xcite . stage 3: each of the corpus-adapted prompts from stage 2 is used to generate a unique set of z queries with flan-t5 xxl (chung et al., 2022). the gold passage for each synthetic query is the passage it was generated from. we have the option of letting z be large because using flan-t5 xxl is considerably less expensive than using gpt-3 as we did in stage 1. in our experiments, we test 1k, 10k, 100k, and 1m as values for z. we primarily focus on @xmath0.3. we use multiple corpus-adapted prompts to mitigate edge cases in which we create a low-quality prompt based on the chosen synthetic queries and in-domain passages from the target domain t . (see stage 4 below for a description of how low-quality prompts can optionally be detected and removed.) as a quality filter for selecting synthetic queries, we test whether a synthetic query can return its gold passage within the top 20 retrieved results using a zero-shot colbertv2 retriever. we only use synthetic queries that pass this filter, which has been shown to improve domain adaptation in neural ir @xcite . stage 4: with each set of synthetic queries generated using the y corpus-adapted prompts in stage 3, we train an individual passage reranker from scratch for the target domain t . these will be used as teachers for a single colbertv2 model in stage 5. our multi-reranker strategy draws inspiration from @xcite , who found a teacher ensemble effective for knowledge distillation into retrievers. at this stage, we can simply use all y of these rerankers for the distillation process. as an alternative, we can select the n best rerankers based on their accuracy on the validation set of the target domain. for our main experiments, we use all of these rerankers. this is the most general case, in which we do not assume that a validation set exists for the target domain. (in appendix a, we evaluate settings where a subset of them is used.) stage 5: the domain-specific passage rerankers from stage 4 serve as multi-teachers creating a single colbertv2 retriever in a multi-teacher distillation process. for distillation, we use annotated triples that are created by using the trained domainspecific reranker to label additional synthetic questions. overall, our distillation process allows us to improve the retrieval accuracy of colbertv2 without needing to use the rerankers at inference. ## discussion & future work our experiments with udapdr suggest several directions for future work: for each dataset, the highest-accuracy zero-shot result is marked in bold while the highest overall is underlined. for udapdr, we use 5 rerankers and 20k distinct synthetic queries for training each of them. the promptagator and gpl results are taken directly from their respective papers. for promptagator, we include both the best retriever-only configuration (promptagator few-shot) and the best retriever + reranker configuration (promptagator++ few-shot). we include the gpl+tsdae pretraining strategy, which is found to improve retrieval accuracy @xcite . we copy the results for bm25, genq, roberta @xcite . we would be interested to see the efficacy of our domain adaptation strategy with these larger encoders. ## conclusion we present udapdr, a novel strategy for adapting retrieval models to new domains. udapdr uses synthetic queries created using generative models, such as gpt-3 and flan-t5 xxl, to train multiple passage rerankers on queries for target domain passages. these passage rerankers are then distilled into colbertv2 to boost retrieval accuracy while keeping query latency competitive as compared to other retrieval systems. we validate our approach across the lotte, beir, nq, and squad datasets. additionally, we explore various model configurations that alter the generative models, prompting strategies, retriever, and passage rerankers used in our approach. we find that udapdr can boost zero-shot retrieval accuracy on new domains without the use of labeled training examples. we also discuss several directions for future work.