To Generate or to Retrieve? On the Effectiveness of Artificial Contexts for Medical Open-Domain Question Answering (2024)

Abstract: Medical open-domain question answering demands substantial access to specialized knowledge. Recent efforts have sought to decouple knowledge from model parameters, counteracting architectural scaling and allowing for training on common low-resource hardware. The retrieve-then-read paradigm has become ubiquitous, with model predictions grounded on relevant knowledge pieces from external repositories such as PubMed, textbooks, and UMLS. An alternative path, still under-explored but made possible by the advent of domain-specific large language models, entails constructing artificial contexts through prompting. As a result, “to generate or to retrieve” is the modern equivalent of Hamlet’s dilemma. This paper presents MedGENIE, the first generate-then-read framework for multiple-choice question answering in medicine. We conduct extensive experiments on MedQA-USMLE, MedMCQA, and MMLU, incorporating a practical perspective by assuming a maximum of 24GB VRAM. MedGENIE sets a new state-of-the-art in the open-book setting of each testbed, allowing a small-scale reader to outcompete zero-shot closed-book 175B baselines while using up to 706x fewer parameters. Our findings reveal that generated passages are more effective than retrieved ones in attaining higher accuracy.

## introduction question answering is a challenging task that requires complex reasoning on explicit constraints described in the question and unstated domain knowledge. open-domain question answering (odqa) aims to tackle natural questions across various topics without predefined evidence @xcite . this setting mirrors real-world scenarios where there cannot be a labeled passage for every potential user inquiry. odqa has particular significance in medicine due to the high-quality standards it demands, including a deep understanding of specialized terminology and background concepts, and an effective recall of expert insight for clinical decision making @xcite . recent efforts have transitioned from a closedbook strategy, where models rely solely on their opaque parametric knowledge, to an open-book alternative, allowing them to consult external sources for grounding. in particular, the retrieve-thenread framework is a common thread @xcite , where the input is augmented with relevant knowledge chunks retrieved from an external datastore, which can be unstructured (e.g., pubmed, textbooks) or structured (e.g., umls). however, performance is highly depen-dent on the quality of the retriever. developing custom retrieval modules generally requires extensive question-context pairs @xcite or intensive computational resources @xcite , particularly when dealing with massive sources @xcite . furthermore, the retrieved fragments may be incomplete or not specifically tailored to the query, leading to noise @xcite . in parallel, medical large language models (llms) have gained increasing research interests to aid professionals and improve patient care @xcite . after pre-training on an extreme-scale collection of specialized text corpora, they implicitly encode an impressive amount of domain knowledge that can be evoked through prompting, akin to summoning a genie from a lamp. this facilitates a paradigm shift towards a generate-then-read approach, wherein contexts are directly generated by an llm. despite preliminary work @xcite @xcite , there is an ongoing debate on "whether generative augmentation is preferable to retrieval augmentation". in this paper, we introduce medgenie, the first generate-then-read framework for multiplechoice medical odqa. specifically, we study the effectiveness of grounding generalist llms and small language models (slms) on multi-view contexts generated by a medical llm via in-context learning (icl) and fine-tuning, respectively. to foster accessibility and match prevalent hardware configurations, we assume a low-resource infrastructure with 24gb vram. we evaluate medgenie on three standard odqa benchmarks designed to quantify professional medical competencies: medqa-usmle @xcite , medmcqa @xcite , and mmlu-medical @xcite . med-genie demonstrates significant performance gains, improving the accuracy of few-shot llm readers on all testbeds by up to ≈16 points. by fine-tuning the reader, medgenie allows flan-t5-base to outcompete closed-book zero-shot 175b llms and supervised 10b baselines on medqa (figure 1 ), using up to 706× fewer parameters. furthermore, our research demonstrates a clear inclination of cutting-edge rerankers towards favoring generated contexts over retrieved ones. when treated as knowledge sources or incorporated into humancurated ones, generated passages notably enhance the effectiveness of retrieve-then-read workflows (up to ≈6 extra points). ragas evaluation con-firms the quality of generated contexts, even allowing for more faithful answers from the reader. finally, we release a comprehensive dataset of ≈1 million artificial contexts in the medical field, adhering to principles of open science and encouraging further research endeavors. open-book question answering closed-book llms, such as gpt-iv and med-palm-2 540b, exhibit impressive results in odqa with fine-tuning @xcite or prompting @xcite techniques, performing non-trivial reasoning steps @xcite . as llms grow to achieve predictable accuracy gains @xcite , their computational demands exceed the resources of most product teams. to counteract the scaling trend, reach satisfactory results with commodity hardware, and ensure greater control over knowledge, various open-book contributions have surfaced in medicine, comprising bioreader @xcite , @xcite , and vod @xcite . nevertheless, to date, generate-then-read pipelines have predominantly undergone scrutiny solely within the general domain. genread @xcite and cgap @xcite are among the first to demonstrate the efficacy of llms as strong context generators, focusing on datasets like natural questions, triviaqa, and webquestions. they utilized llms such as instructgpt (175b) and megatron (530b) both as generators and readers. genread introduced a clustering-based prompting technique involving context generation, filtering, encoding, kmeans clustering, and per-cluster sampling. while we acknowledge the efficacy of these techniques, as demonstrated in ad-hoc ablations (appendix d), we caution against their feasibility in real-world scenarios due to associated high costs. additionally, investigations into the fusion of retrieved and llm-generated contexts for odqa, exemplified by combo @xcite , have shown promising results in enhancing performance across aforementioned benchmarks. however, their approach entails utilizing two discriminators trained on silver labels to compute compatibility scores, with instructgpt and chatgpt as context generators and a fid flan-t5-large as the reader, scaling up to 10 contexts using four a40 gpus, each equipped with 46 gb vram. ## method in this section, we discuss our medgenie framework, illustrated in figure 2 . first, it prompts a medical llm to furnish multi-view background contexts for a given question ( §3.1). then, it diverges into two distinct paths depending on the reader type: icl for llms ( §3.2), and fusion-in-decoder (fid) fine-tuning for slms ( §3.3). both strategies equip readers with custom background passages, allowing them to tackle medical questions effectively even without prior knowledge. problem statement in multiple-choice odqa, a dataset record consists of a question q and an answer set @xmath0, . . . , a |a| }, all expressed in text. our goal is to identify which answer @xmath1. a closed-book solution would estimate p θ (a|q, a) parameterized by θ, leaving considerable domain knowledge unexploited @xcite . conversely, we assume access to a set of record-specific artificial contexts and add an auxiliary variable @xmath2, . . . , c |c| }. we thus move to a marginal likelihood with a reader-generator model: p θ (a|q, a)= i p θ (a|c i , q, a)p θ (c i |q, a). since we cannot sum over all possible contexts, we approximate the maximum a posteriori estimation with a decoding strategy, using a single value for c. ## results our core findings are delineated in