A Thorough Examination on Zero-shot Dense Retrieval (2023)

Abstract: Recent years have witnessed the significant advance in dense retrieval (DR) based on powerful pre-trained language models (PLM). DR models have achieved excellent performance in several benchmark datasets, while they are shown to be not as competitive as traditional sparse retrieval models (e.g., BM25) in a zero-shot retrieval setting. However, in the related literature, there still lacks a detailed and comprehensive study on zero-shot retrieval. In this paper, we present the first thorough examination of the zero-shot capability of DR models. We aim to identify the key factors and analyze how they affect zero-shot retrieval performance. In particular, we discuss the effect of several key factors related to source training set, analyze the potential bias from the target dataset, and review and compare existing zero-shot DR models. Our findings provide important evidence to better understand and develop zero-shot DR models.

## introduction with the massive success of pre-trained language models (plm) @xcite , dense retrieval (dr) has been empowered to become an essential technique in first-stage retrieval. instead of using sparse term-based representations, dr learns low-dimensional query and document embeddings for semantic matching, which has been shown to be competitive with or even better than sparse retrievers (e.g., bm25) @xcite . unlike sparse retrievers, dr models typically require training on sufficient labeled data to achieve good performance in a specific domain. however, there are always new domains (or scenarios) that need to be dealt with for an information retrieval system, where little in-domain labeled data is accessable. therefore, it is crucial * the work was done during the internship at baidu. † corresponding authors. to investigate the retrieval capabilities of dr models under the zero-shot setting, where dr models trained on an existing domain (called source domain) is directly applied to another domain (called target domain) with no available training data on the target domain. considering the large discrepancy between different domains, the zero-shot capability directly affects the widespread deployment of dr models in real-world applications. recently, a number of studies have been conducted to analyze the zero-shot capability of dr models @xcite , and report that dr models have poor zero-shot retrieval performance when compared with lexical models such as bm25. however, existing works primarily focus on zero-shot performance across a wide range of tasks, and lacks a comprehensive and in-depth analysis on the influence of different settings affected by various factors. thus, the derived findings may not hold under some specific settings, which cannot ensure an accurate understanding of the zero-shot retrieval capacity. for example, our empirical analysis has revealed that bm25 is not the absolute leader compared with dr models under the zero-shot setting, and different source training sets result in different zero-shot performances (section 2.3). as a result, a thorough examination of the zero-shot capacity of dr models is necessary. considering these issues, this paper aims to provide an in-depth empirical analysis of zero-shot dr. specifically, we would like to conduct a more detailed analysis by analyzing the effect of different factors on retrieval performance. since the dr models are trained on the source domain data (called source training set), we focus on the analysis by examining three major aspects of the source training set, including query set, document set, and data scale. besides, we also consider other possible factors such as query type distribution, vocabulary overlap between source and target sets, and the bias of the target dataset. our analysis is particularly interested in answering the research questions of what influence factors affect the zero-shot capabilities of dr models and how does each of the influence factors affect such capability? in addition, we also systematically review the recent advances in zero-shot dr. we summarize and categorize the key techniques that can improve the zero-shot performance (with an associated discussion of the potential influence factors). by our empirical analysis, we find that: (i) training data from source domain has important effect on the zero-shot capability of the trained dr model, either the query set or document set in it (section 3.1 and section 3.2). (ii) vocabulary overlap and query type distribution are potential factors that affect the zero-shot performance (section 3.1.2 and section 3.1.3). (iii) increasing the query scale of source data can improve both the in-domain and out-of-domain performance of the dr model (section 3.3.1). (iv) increasing the document scale of the source domain may result in performance decrease (section 3.3.2). (v) the lexical bias (overlap coefficient of queries and annotated documents) of some datasets is probably one of the reasons that the performance of sparse retriever seems to be more robust than dr model on existing benchmarks (section 3.4). our study provides an important work to understand and improve the zero-shot capacity of the dr model. ## experimental analysis and findings in this section, we conduct a detailed analysis about multiple factors of the source training set, including source query set, source document set, and data scale. besides, we also analyze the effect of the bias from the target dataset. ## model analysis so far, we mainly focus on the discussions of data side. in this section, we further review and analyze existing zero-shot dr models. ## conclusion and future work in this paper, we thoroughly examine the zero-shot capability of dr models. we conduct empirical analysis by extensively studying the effect of various factors on the retrieval performance. in particular, we find that the factors of vocabulary overlap, query type distribution, and data scale are likely to have varying degrees of influence on the zeroshot performance of the dense retriever. besides, the performance between bm25 and dr models varies significantly on different target datasets, where the dataset bias (e.g., a dataset is created based on exact match) is likely to make such comparison unfair. overall, we find that the zero-shot performance of dense retrieval models still has room to improve and deserves further study. as future work, we will consider designing more robust and general zero-shot retrieval methods that can adapt to different settings or domains.