A uto MIR : Effective Zero-Shot Medical Information Retrieval without Relevance Labels (2025)

Abstract: Medical information retrieval (MIR) is vital for accessing knowledge from electronic health records, scientific literature, and medical databases, supporting applications such as medical education, patient queries, and clinical diagnosis. However, effective zero-shot dense retrieval in the medical domain remains difficult due to the scarcity of relevance-labeled data. To address this challenge, we propose **S**elf-**L**earning **Hy**pothetical **D**ocument **E**mbeddings (**SL-HyDE**), a framework that leverages large language models (LLMs) to generate hypothetical documents conditioned on a query. These documents encapsulate essential medical context, guiding dense retrievers toward the most relevant results. SL-HyDE further employs a self-learning mechanism that iteratively improves pseudo-document generation and retrieval using unlabeled corpora, eliminating the need for labeled data. In addition, we introduce the Chinese Medical Information Retrieval Benchmark (CMIRB), a comprehensive evaluation suite reflecting real-world medical scenarios, comprising five tasks and ten datasets. By benchmarking ten models on CMIRB, we provide a rigorous standard for evaluating MIR systems. Experimental results demonstrate that SL-HyDE significantly outperforms HyDE in retrieval accuracy, while exhibiting strong generalization and scalability across diverse LLM and retriever configurations. Our code and data are publicly available at: https://github.com/ll0ruc/AutoMIR.

## introduction medical information retrieval (mir) @xcite focuses on retrieving relevant medical information from sources like electronic health records, scientific papers, and medical knowledge databases, based on specific medical queries. its applications are wide-* xiao zhou and zheng liu are corresponding authors. dense retrievers @xcite have shown strong performance when trained on large labeled datasets in information retrieval (ir). several studies @xcite @xcite have successfully employed contrastive learning to develop general-purpose text embedding models, achieving promising results in zero-resource retrieval scenarios. they leverage large-scale weakly supervised data through web crawling as well as high-quality text pairs derived from data mining or manual annotation. however, the availability of such large-scale datasets cannot always be guaranteed, particularly in non-english languages or specialized domains. recently, large language models (llms) have demonstrated exceptional performance in zeroresource retrieval scenarios @xcite , primarily due to their extensive knowledge and powerful text generation capabilities. this makes them particularly effective in situations where labeled data are scarce or unavailable. one such approach, hyde @xcite , employs zero-shot prompts to guide an instructionfollowing language model to generate hypothetical documents, effectively narrowing the semantic gap between the query and the target document. similarly, query2doc @xcite uses few-shot prompting of llms to generate pseudo-documents, which are then used to augment the original query. however, applying these hyde-style methods to medical information retrieval presents three critical challenges: (1) llms lack the specialized medical knowledge necessary to generate highly relevant hypothetical documents. hyde employs general-purpose llms for pseudo-document generation, which are often insufficiently equipped with domain-specific knowledge, particularly in fields like medicine. community efforts such as huatuogpt @xcite and pmc-llama @xcite highlight the necessity of fine-tuning on medical data to enhance domain-specific capabilities. although these medical llms have richer medical knowledge, their outputs are suboptimal in aligning with retrieval optimization needs. (2) general text embedding models are inadequate for representing medical queries and documents effectively. these versatile retrievers @xcite are typically designed for multi-domain and multitask settings, but fail to capture the nuanced and knowledge-intensive nature of the medical domain. to address these issues, we propose self-learning hypothetical document embedding (sl-hyde), an effective fully zero-shot dense retrieval system that requires no relevance-labeled data for medical information retrieval. during the inference phase, sl-hyde first employs an llm as the generator to produce a relevant hypothetical document in response to a medical query. a retrieval model is then used to identify the most relevant target document from the candidates based on the generated hypothetical document. in the training phase, we design a self-learning mechanism that enhances the retrieval performance of sl-hyde without the need for labeled data. specifically, this mechanism leverages the retrieval model's ranking capabilities to select high-relevance hypothetical documents that align with the outputs of the generator (llm), simultaneously injecting medical knowledge into the llm. in turn, the generator's ability to produce high-quality hypothetical documents provides pseudo-labeled data for the training of the retrieval model, enabling it to efficiently encode medical texts. this interactive and complementary approach generates supervisory signals that enhance both the generation and retrieval capabilities of the system. notably, sl-hyde begins with unlabeled medical corpora and completes the training process through a self-learning mechanism, thereby circumventing the heavy reliance on labeled data typically required for training both large language models and text embedding models. to evaluate sl-hyde's performance in chinese medical information retrieval, we develop a valuable chinese medical information retrieval benchmark (cmirb). cmirb is constructed from real-world medical scenarios, including online consultations, medical examinations, and literature retrieval. it comprises five tasks and ten datasets, representing the first comprehensive and authentic evaluation benchmark for chinese medical information retrieval. this benchmark is expected to accelerate advancements toward building more robust and generalizable mir systems in the future. through extensive experimentation on cmirb, we find that our proposed method significantly enhances retrieval performance. we validate sl-hyde across various configurations involving three large language models as generators and three embedding models as retrievers. notably, sl-hyde surpasses the hyde (qwen2 as generator + bge as retriever) combination by an average of 4.9% in ndcg@10 across ten datasets, and achieves a 7.2% improvement compared to using bge alone for retrieval. these outcomes underscore the effectiveness and versatility of sl-hyde. in summary, our contributions are as follows: ## conclusions in this paper, we propose sl-hyde, an automated framework for zero-shot medical information retrieval that operates without reliance on labeled relevance data. leveraging an unlabeled medical corpus, sl-hyde employs a self-learning training paradigm where the retriever guides the generator's training, and the generator in turn produces pseudodocuments that enhance retriever training. this process injects domain-specific medical knowledge into both components, yielding hypothetical documents that are highly effective in guiding retrieval. furthermore, we introduce cmirb, a comprehensive benchmark for chinese medical information retrieval, encompassing five tasks and ten datasets. extensive experiments demonstrate that sl-hyde consistently outperforms hyde across all datasets. additionally, sl-hyde shows strong adaptability and scalability, effectively enhancing retrieval performance across various combinations of generators and retrievers. in future work, we plan to extend sl-hyde to other data-scarce domains to further evaluate its generalizability across different settings. in addition, we will explore reinforcement learning techniques to further enhance retriever capabilities and improve reasoning in complex medical retrieval scenarios.