Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? (2024)

Abstract: Several recent works seek to develop foundation models specifically for medical applications, adapting general-purpose large language models (LLMs) and vision-language models (VLMs) via continued pretraining on publicly available biomedical corpora. These works typically claim that such domain-adaptive pretraining (DAPT) improves performance on downstream medical tasks, such as answering medical licensing exam questions. In this paper, we compare seven public “medical” LLMs and two VLMs against their corresponding base models, arriving at a different conclusion: all medical VLMs and nearly all medical LLMs fail to consistently improve over their base models in the zero-/few-shot prompting regime for medical question-answering (QA) tasks. For instance, across the tasks and model pairs we consider in the 3-shot setting, medical LLMs only outperform their base models in 12.1% of cases, reach a (statistical) tie in 49.8% of cases, and are significantly worse than their base models in the remaining 38.2% of cases. Our conclusions are based on (i) comparing each medical model head-to-head, directly against the corresponding base model; (ii) optimizing the prompts for each model separately; and (iii) accounting for statistical uncertainty in comparisons. While these basic practices are not consistently adopted in the literature, our ablations show that they substantially impact conclusions. Our findings suggest that state-of-the-art general-domain models may already exhibit strong medical knowledge and reasoning capabilities, and offer recommendations to strengthen the conclusions of future studies.

## introduction recent advances in autoregressive large language models (llms) and vision-language models (vlms) have attracted interest from practitioners in medicine, where these models hold great potential to transform various aspects of clinical practice (e.g., medical diagnosis, information retrieval from clinical documents, patient triaging) @xcite . state-of-the-art performance on various medical benchmarks is typically achieved by massive-scale closed-source models, such as @xcite , med-gemini @xcite , and med-palm @xcite , often performing on par with humans on medical licensing exams and open-ended consumer health question-answering (qa) tasks. however, the general lack of transparency in these models, high api usage costs, and patient data privacy concerns make their integration into routine clinical workflows challenging @xcite . to address such concerns, recent works have proposed cheaper, open-source alternatives through domain-adaptive pretraining (dapt; @xcite , where a pretrained open-source general-domain model-such as llama @xcite or mistral @xcite in the language space, and llava @xcite or @xcite in the vision-language space-is continually pretrained on biomedical (image-)text corpora from public sources such as pubmed and medical textbooks. while some prior works show that medical models pretrained from scratch only using domain-specific corpora can outperform those trained via dapt, both in the context of bertstyle encoder-only models @xcite @xcite and decoder models @xcite @xcite , the dapt approach has become common practice, resulting in a trend where the release of a more capable generaldomain model is typically followed by the release of its medical counterpart. despite the widespread adoption of medical dapt, the claimed improvements in performance are worth scrutinizing. while the story is intuitive, more recent base models (e.g., llama-3-8b (meta, 2024)) already exhibit strong off-theshelf performance on medical benchmarks without any adaptation (e.g., open medical llm leaderboard @xcite ), and given a lack of transparency about the pretraining corpora used to train the general-domain model in the first place, they may already be trained on relevant medical text. perhaps more concerning is the lack of apples-toapples comparisons in the literature. first, medical models resulting from dapt are often only compared against other baselines with different architectures (e.g., clinicalcamel-70b @xcite vs. @xcite ) and under inconsistent evaluation setups (e.g., @xcite fine-tuned on medqa @xcite vs. non-fine-tuned med42-v1-70b @xcite ), which can confound the interpretation of results. second, the common practice of using a single, fixed prompting setup (e.g., prompt format, choice of few-shot examples) for all models under evaluation also warrants concern, as llm/vlm behavior is extremely sensitive to such design decisions @xcite @xcite , and the "optimal" choice of such details rarely correlates between different models @xcite . in this paper, we perform an apples-to-apples comparison that addresses these concerns, comparing seven medical llms and two medical vlms against their general-domain base models. we find that, for all but one llm pair-biomistral-7b @xcite vs. mistral-7b-instruct-v0.1 @xcite , a pair of models that performs fairly poorly in absolute terms-the open-source medical llms and vlms that we evaluate do not consistently improve over their general-domain counterparts on various medical (visual) qa tasks (figure 1 ). we compare several pairs of general-domain and medically adapted llms/vlms (see our findings (section 4) suggest that state-ofthe-art general-domain models may already exhibit strong medical knowledge and reasoning capabilities that can be leveraged effectively when prompted appropriately. our main contributions can be summarized as follows: ## related work dapt @xcite ) is a transfer learning approach, where a pretrained model is further pretrained on domain-specific data for better alignment to a target domain of interest (e.g., medicine, law). several studies show that language models trained via dapt often outperform their generaldomain counterparts on domain-specific tasks, such as claim detection from blog posts @xcite , named entity recognition from german novels (konle and jannidis, 2020), and judgment prediction for legal cases @xcite . in the medical domain, prior works based on bertstyle encoder-only language models @xcite , such as biobert @xcite and @xcite , show that medical dapt improves fine-tuning performance on tasks such as medical concept extraction from patient reports @xcite , identification of gene-disease relations from pubmed abstracts @xcite @xcite , and natural language inference on clinical notes @xcite . more recent works suggest that decoder-based autoregressive llms and vlms trained via medical dapt also show strong performance on various medical tasks. medical llms such as med-itron @xcite , adapted from llama-2 @xcite ; and biomistral @xcite , adapted from mistral-7b-instruct-v0.1 @xcite ; perform well on knowledge-intensive qa tasks based on medical licensing and academic exams @xcite @xcite and pubmed abstracts @xcite . medical vlms such as llava-med @xcite , adapted from llava @xcite ; and med-flamingo @xcite , adapted from open-flamingo @xcite ; also perform well on visual qa tasks based on radiology @xcite and pathology images @xcite and academic exams @xcite . these encouraging results have established dapt as a go-to approach for training a medically specialized model, a conclusion that we re-examine in this work. ## experimental setup to investigate the effectiveness of medical dapt in improving zero-/few-shot performance, we compare 7 medical llms and 2 medical vlms against their general-domain counterparts in pairs (figure 1(a) ), on 13 textual qa datasets and 8 visual qa datasets, respectively. the models in each pair are exactly identical in model architecture and scale, and their only difference lies in whether they were additionally pretrained on medical data. we also note that while some of datasets used for evaluation contain both closed-ended (i.e., has clear groundtruth answers) and open-ended questions, we focus our evaluations on the former, where an objective, quantitative assessment of medical knowledge and reasoning capabilities is possible. for reproducibility of our results, we open-source the source code used for all of our evaluations described below via our github repository foot_0 . ## discussion and conclusion in this work, we investigated the effectiveness of dapt for training medically specialized llms and autoregressive vlms suitable for knowledgeintensive medical (visual) qa tasks. to that end, we compared several pairs of state-of-the-art medical llms/vlms to their general-domain counterparts, whose only differences lie in medical dapt and are exactly identical in model architecture and scale. our work diverges from prior works by providing a direct apples-to-apples comparison of medical and general-domain models while accounting for llm/vlm sensitivity to prompting details and assessing the statistical significance of the results. across both model classes and all model scales, we found that the performance benefits from medical dapt largely disappear when we (i) tailor the prompt format and choice of few-shot examples to each medical and general-domain model separately; and (ii) account for statistical uncertainty in model comparison. in particular, we found that when we optimize the prompt only for the medical model and compare each model pair based on their absolute accuracies without accounting for uncertainty, the performance improvements from medical dapt can be overestimated, potentially leading to unreliable conclusions about the benefits of medical dapt. for example, in the zero-shot setting, evaluation under this setup leads to the conclusion that medical llms and vlms, on average, outperform the corresponding general-domain models in 70.5% and 62.5% of all qa tasks, while the improvements are in reality statistically significant in only 9.4% and 6.3% of tasks after optimizing the prompt for each model to ensure a fair comparison. our findings suggest that for state-of-the-art general-domain llms and vlms, the performance benefits from additionally pretraining on medical data from public sources such as pubmed may be limited. notably, almost all of the medical models used in our evaluation use pubmed as the primary source of pretraining data for medical adaptation (