Precise Zero-Shot Dense Retrieval without Relevance Labels (2023)

Abstract: While dense retrieval has been shown to be effective and efficient across tasks and languages, it remains difficult to create effective fully zero-shot dense retrieval systems when no relevance labels are available. In this paper, we recognize the difficulty of zero-shot learning and encoding relevance. Instead, we propose to pivot through Hypothetical Document Embeddings (HyDE). Given a query, HyDE first zero-shot prompts an instruction-following language model (e.g., InstructGPT) to generate a hypothetical document. The document captures relevance patterns but is “fake” and may contain hallucinations. Then, an unsupervised contrastively learned encoder (e.g., Contriever) encodes the document into an embedding vector. This vector identifies a neighborhood in the corpus embedding space, from which similar real documents are retrieved based on vector similarity. This second step grounds the generated document to the actual corpus, with the encoder’s dense bottleneck filtering out the hallucinations. Our experiments show that HyDE significantly outperforms the state-of-the-art unsupervised dense retriever Contriever and shows strong performance comparable to fine-tuned retrievers across various tasks (e.g. web search, QA, fact verification) and in non-English languages (e.g., sw, ko, ja, bn).

## introduction dense retrieval @xcite , the method of retrieving documents using semantic embedding similarities, has been shown to be successful across tasks like web search, question answering, and fact verification. a variety of methods such as negative mining @xcite , distillation @xcite @xcite , retrievalspecific pre-training @xcite @xcite @xcite and scaling @xcite have been proposed to improve the effectiveness of supervised dense retrieval models. nevertheless, zero-shot dense retrieval still remains difficult. many recent works consider the alternative transfer learning setup, where the dense retrievers are trained on a high-resource dataset and then evaluated on queries from different domains. ms marco @xcite , a dataset with a large number of manually judged query-document pairs, is the most commonly used. as argued by @xcite , in practice, however, the existence of such a large dataset cannot always be assumed. furthermore, ms marco restricts commercial use and cannot be adopted in a variety of real-world search scenarios. in this paper, we aim to build effective fully zero-shot dense retrieval systems that require no relevance supervision, work out-of-box and generalize across emerging search tasks. as supervision is not available, we start by examining self-supervised representation learning methods. modern deep learning enables two distinct approaches. at the token level, generative large language models (llms) pre-trained on large corpora have demonstrated strong natural language understanding (nlu) and generation (nlg) capabilities @xcite @xcite @xcite . at the document level, text (chunk) encoders pre-trained with contrastive objectives learn to encode documentdocument similarity into inner products @xcite . on top of these, one extra insight from llms is borrowed: llms further trained to follow instructions can zero-shot generalize to diverse unseen instructions @xcite @xcite . in particular, in-structgpt shows that with a small amount of data, ## methodology in this section, we first formally define the problem of (zero-shot) dense retrieval. then we will introduce how hyde is designed to solve it. ## experiments in this section, we discuss how we implement hyde and test it as a zero-shot out-of-box search system. we show how much hyde improves over the base unsupervised dense encoder as well as how it compares to models with rich supervision. ## analysis the generative llm and contrastive encoder make up the two core components of hyde. in this section, we study the effects of changing their realizations. in particular, we consider smaller language models (lms), lms without instruction following and fine-tuned encoders. we also demonstrate a way to visualize and better understand hyde. ## conclusion in this paper, we introduce hyde, a new approach for building effective dense retrievers in a completely unsupervised manner, without the need for any relevance labels. we demonstrate that some aspects of relevance modeling can be delegated to a more powerful, flexible, and general-purpose llm that has not specifically been adapted for search tasks. as a consequence, the need for relevance labels is eliminated, replaced by pure generation. we are excited to see if this can be generalized further to more sophisticated tasks like multi-hop retrieval/qa and conversational search. despite its dependence on llms, we argue that hyde is of practical use in real-world applications, though not necessarily over the entire lifespan of a search system. at the very beginning of building a search system, serving queries using hyde offers performance comparable to a fine-tuned model, which no other relevance-free model can offer. as search logs grow and relevance data accumulate, a supervised dense retriever can be gradually trained and then rolled out. as the dense retriever becomes more capable, it can handle queries that are "indomain", while hyde can remain useful for novel, unexpected, or emerging queries.