Does Biomedical Training Lead to Better Medical Performance? (2025)

Abstract: Large Language Models (LLMs) hold significant potential for improving healthcare applications, with biomedically adapted models promising enhanced performance on medical tasks. However, the effectiveness of biomedical domain adaptation for clinical tasks remains uncertain. In this study, we conduct a direct comparison of 12 biomedically adapted models and their general-domain base counterparts across six clinical tasks. Our results reveal that 11 out of 12 biomedical models exhibit performance declines, challenging prior findings that reported positive effects of biomedical adaptation. Notably, previous positive results primarily relied on multiple-choice evaluations, which may not reflect performance in real-world clinical applications. To promote reproducibility and further research, we open-source our evaluation pipeline, providing a resource for the development of models with practical benefits in healthcare settings.

## introduction large language models (llms) have the potential to transform healthcare by enhancing patient care quality and efficiency @xcite . opensource biomedical llms, designed for medical applications, promise improved performance with fewer parameters than general models @xcite @xcite . however, recent research questions the effectiveness of biomedical domain adaptation @xcite @xcite . in this study we perform a direct comparison of 12 biomedically adapted models with their generaldomain base models on six clinical tasks. our results reveal performance declines in 11 of 12 biomedical models. this is in contrast to previous findings that reported positive effects of biomedical training @xcite @xcite . however, these studies primarily relied on multiple-choice evaluations that did not incorporate real-world clinical documents. this suggests that the observed benefits of biomedical adaptation may not translate effectively to practical healthcare settings. to facilitate reproducibility and enable future development of models with practical benefits in healthcare settings, we open-source our evaluation pipeline. by providing a standardized framework for assessing biomedical llms on real-world clinical tasks, we aim to bridge the gap between benchmark performance and real-world applicability. ## related work the need for specialized healthcare tools has recently accelerated biomedical llm development, yielding commercial models like med-palm @xcite and medgemini @xcite , and open-source alternatives such as meditron @xcite , biomistral @xcite , internist.ai @xcite , and med42 @xcite . although biomedical llms initially outperformed general-domain models on tasks like multiple-choice question-answering (mcqa) exams, recent studies @xcite @xcite ## evaluation tasks we introduce the clinical language understanding evaluation (clue) consisting of six tasks on clinical notes, consumer health questions, electronic health records (ehr) and discharge summaries, encompassing information extraction, summarization, clinical reasoning, simplification, and coding. we divide the tasks into two levels. level 1 includes simpler tasks with short inputs, while level 2 has complex tasks with long inputs. we provide prompt examples for each task in figures 2, 3, 4, 5, 6, and 7 in appendix b.3. mednli @xcite is based on clinical notes from mimic-iii @xcite . it evaluates models on predicting the logical relationship-contradiction, neutrality, or entailment-between a premise and hypotheses, testing clinical reasoning with short input lengths. meqsum (ben abacha and demner-fushman, 2019) contains 1,000 consumer health inquiries summarized by medical experts. this task evaluates whether models can understand lay language, extract key information, and reformulate patient queries into concise, medically sound questions. problem summary derived from soapstructured clinical notes, this task was first described by @xcite and utilizes the subjective and assessment sections for predicting a patient's health problems @xcite . like mednli, its short input length tests basic information extraction abilities. longhealth @xcite consists of 20 fictional patient records designed to challenge llms on long input comprehension. evaluation involves answering questions on multiple long documents, handling added irrelevant information, and recognizing when data is unavailable. this task assesses comprehension, long-input retention, and hallucination tendencies. medisumqa @xcite requires models to comprehend mimic-iv @xcite discharge summaries, extract key information, answer patient-related queries, and simplify medical information. additionally, models must apply medical knowledge to provide appropriate follow-up advice. using mimic-iv, we create medisumcode, an icd-10 prediction dataset by linking discharge summaries with annotated icd-10 codes via hospital admission ids. this dataset provides discharge summaries as inputs and icd-10 codes as labels for model evaluation. medisumcode involves assigning icd-10 codes to diagnoses and procedures in discharge summaries, a critical task for patient records, billing, and healthcare analysis @xcite . this challenge requires models to extract diagnoses from complex clinical text, comprehend over 70,000 icd-10 codes, and accurately match diagnoses to the correct codes. ## experimental setup we evaluated 24 language models, including biomedically trained models, their base models, and additional general-domain models as reference. our evaluation aims to (1) measure the effects of continuous biomedical training, (2) assess whether biomedical models or general-domain models are more suitable for specific medical scenarios, and (3) rank current openly available models. appendix a.1 describes the metrics we applied to each task. for each task, we report the average over all metrics. ## results most performance gains are observed in models based on llama-2 and mistral-7b-v0.1, while models derived from more recent llms frequently underperform after adaptation. additionally, improvements are more common in models with up to 8b parameters, whereas larger models tend to lose performance after biomedical training. figure 1 shows a comparison between the best-performing biomedical models and their base models. we model mcqa level 1 level 2 meditron-7b +6.07 -7.08 -meditron-70b +3.63 -4.59 -biomistral-7b +4.13 +0.26 +0.71 biomistral-7b-dare +4.57 +2.93 +2.7 internist.ai 7b --2.07 +5.52 openbiollm-8b -0.63 -15.17 -13.54 openbiollm-70b +1.46 -4.78 -10.45 med42-8b +0.47 +2.51 -1.4 med42-70b +2.8 -7.57 -15.14 aloe-8b-alpha +2.21 -5.87 -8.67 meditron3-8b --2.76 -15.04 meditron3-70b --2.18 -8.51 task complexity also plays a key role: gains are mainly seen in level 1 tasks, while performance on more complex level 2 tasks often declines. this suggests biomedical models may strug-gle with tasks requiring language understanding and reasoning. unlike previous reports of biomedical llm improvements on mcqa evaluations, only two models show slight average gains on both level 1 and level 2 tasks on clue (see overall, general-domain llms remain strongest, with llama3.1-70b emerging as the top performer. although llama3-med42-8b slightly outperforms its base model on simple tasks (+0.56%), it shows a large drop on level 2 tasks (-8.03%). ## discussion performance declines are observed across various training methods, except for biomistral-dare, which uses weight merging, indicating a potential mitigation strategy. however, the superior performance of mistral-7b-instruct-v0.2 ( improvements were almost exclusive to the lower-performing mistral-7b-instruct-v0.1 models, suggesting that recent general models like llama-3 and mistral-7b-v0.2 already address these gaps. tables 4 and 3 further support this. ## conclusion our study suggests that biomedical llms are not competing effectively with general-domain models on clinical tasks. while some biomedical models have shown improvements, more recent and larger models are underperforming. fine-tuning these models with domain-specific data often leads to reduced performance, introducing hallucinations and decreased model stability. this stands in contrast to traditional mcqa evaluations, where biomedical models have previously demonstrated superior performance. our evaluation provides a more practical assessment of llm capabilities in real-world healthcare settings. to support further progress in this field, we open-source our evaluation scripts, allowing for broader validation and replication of our results.