Relevance-assisted Generation for Robust Zero-shot Retrieval (2023)

Abstract: Zero-shot retrieval tasks such as the BEIR benchmark reveal out-of-domain generalization as a key weakness of high-performance dense retrievers. As a solution, domain adaptation for dense retrievers has been actively studied. A notable approach is synthesizing domain-specific data, by generating pseudo queries (PQ), for fine-tuning with domain-specific relevance between PQ and documents. Our contribution is showing that key biases can cause sampled PQ to be irrelevant, negatively contributing to generalization. We propose to preempt their generation, by dividing the generation into simpler subtasks, of generating relevance explanations and guiding the generation to avoid negative generalization. Experiment results show that our proposed approach is more robust to domain shifts, validated on challenging BEIR zero-shot retrieval tasks.

## introduction despite strong in-domain performance, dense retrievers have shown poor generalization to out-ofdomain (ood) zero-shot tasks where no training queries are available @xcite . to enable training, pseudo-query generation (pqg) @xcite has shown promising results, by generating in-domain pseudo queries q from a target corpus d. however, we show q are often irrelevant to the documents for which they were generated, and generating a single document vector from the finetuned document encoder using q is often insufficient. figure 1 (a) illustrates the two limitations of the standard pqg approach, and figure 1 (b) our solutions, discussed as follows. to tackle the two limitations, we propose relevance-assisted multi-query domain adaptation, or ramda 1 . first, for relevance-zero-shot task such as the beir benchmark reveals the key weakness of high-performance dense retrievers @xcite zero-shot task such as the beir benchmark reveals the key weakness of high-performance dense retrievers @xcite how to improve performance? we conduct experiments on beir benchmarks, which include diverse out-of-domain retrieval tasks. the results show that, compared to the baseline pqg, our proposed ramda increases ndcg@10 and recall@100, by 2.4 pt and 4.6 pt on average, respectively. further analyses show that our generated queries approximate gold queries better, and capture diverse queries. our contributions are threefold: ## related work to address retrieval across diverse domains, dense retrievers have been trained using opendomain large-scale corpus, in supervised manner @xcite @xcite when relevance annotations are available (e.g., ms marco @xcite ), or using self-supervised learning in cases where such annotations are absent @xcite . however, dense retrievers have shown poor performance when tested on specialized out-of-domain datasets, due to distribution shifts @xcite . towards improved generalization, we discuss two approaches that tackle the challenge of distribution shifts: 1) improving training and 2) robustifying inference. ## method given a document d in a target corpus d, pqg aims to generate pseudo-queries @xmath0, as alternatives to gold queries q d . following previous work, we employ t5 @xcite as the backbone generator. ## conclusion we investigated pqg for overcoming domain shifts in zero-shot retrieval, motivated by the observation that generated pqs often negatively affect such a goal. we show term frequency and diversity bias as a cause, and propose a novel pqg method that preempts negative pqg. we validate with extensive experiments on the beir benchmark, that through relevance-guidance and multi-query generation, our proposed model effectively addresses the challenges of domain shifts in zero-shot retrieval.