LLME val- M ed: A Real-world Clinical Benchmark for Medical LLM s with Physician Validation (2025)

Abstract: Evaluating large language models (LLMs) in medicine is crucial because medical applications require high accuracy with little room for error. Current medical benchmarks have three main types: medical exam-based, comprehensive medical, and specialized assessments. However, these benchmarks have limitations in question design (mostly multiple-choice), data sources (often not derived from real clinical scenarios), and evaluation methods (poor assessment of complex reasoning). To address these issues, we present LLMEval-Medicine, a new benchmark covering five core medical areas, including 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios. We also design an automated evaluation pipeline, incorporating expert-developed checklists into our LLM-as-Judge framework. Furthermore, our methodology validates machine scoring through human-machine agreement analysis, dynamically refining checklists and prompts based on expert feedback to ensure reliability. We evaluate 13 LLMs across three categories (specialized medical models, open-source models, and closed-source models) on LLMEval-Med, providing valuable insights for the safe and effective deployment of LLMs in medical domains.

## introduction language model based ai systems demonstrate significant potential in medical applications. these technologies are rapidly transforming healthcare delivery and decision support. examples include dataset open qa closed qa know. reason. ethics medexam (2025) ✗ ✓ ✓ ✗ ✓ medqa (2023) ✗ ✓ ✓ ✗ ✗ medmcqa (2022) ✗ ✓ ✓ ✓ ✗ pubmedqa (2019) ✗ ✓ ✓ ✓ ✗ mednli (2021) ✗ ✗ ✗ ✓ ✗ medcalc (2024) ✗ ✓ ✗ ✗ ✗ multimedqa (2022) ✓ ✓ ✓ ✓ ✗ medbench (2023) ✓ ✓ ✓ ✓ ✗ medjourney (2024) ✓ ✓ ✗ ✓ ✗ llmeval-med (ours) ✓ ✓ ✓ ✓ ✓ cleveland clinic's watson chatbots for disease management (clinic, 2016), google's med-palm2 for diagnostic suggestions @xcite , and stanford's simulation systems for medical education @xcite . however, these medical ai systems require exceptional accuracy as errors directly impact patient safety. several concerning cases highlight this challenge: ibm's watson for oncology recommended unsafe cancer treatments @xcite ; ai systems showed higher false-positive rates than radiologists in lung disease detection @xcite ; and diabetic retinopathy screening tools demonstrated inconsistent performance with sensitivities ranging from 51% to 86% @xcite . these examples emphasize the critical need for rigorous testing before deploying llm-powered ai in high-risk medical environments, making the development of robust benchmarks an extremely important issue. to date, various benchmarks have been proposed and can be grouped into three main categories: exam-based benchmarks, such as aiims (india) ## related work in recent years, large-scale question-answering benchmarks have formed the backbone of medical llm evaluation. medmcqa @xcite and medqa @xcite draw on thousands of indian postgraduate and usmle (2025) exam items to test factual recall and domain reasoning, while multimedqa @xcite unifies diverse sources-from pubmed @xcite abstracts to consumer health queries-to challenge models across both specialist and layperson prompts. beyond simple recall, specialized tests probe inferential and quantitative skills. mednli @xcite reframes clinical note pairs as entailment tasks to assess models' ability to infer diagnoses and contradictions, and medcalc-bench @xcite offers patient-scenario cal-culations, demanding both correct numerical output and clear stepwise justification. to mirror real-world practice and multimodal demands, benchmarks like medjourney @xcite trace the full care pathway-from initial planning through follow-up-while worldmedqa-v @xcite pairs multilingual exam questions with medical images, testing cross-lingual and vision-language capabilities in four languages. for mandarin-language evaluation, webmedqa @xcite supplies over 60,000 real patient questions with verified answers; cmedqa @xcite and its successor cmedqa2 @xcite draw on community forums to produce 50,000-100,000 clinician-patient exchanges; and cblue @xcite spans eight nlu tasks-named entity recognition, relation extraction, diagnosis normalization, and more-to reveal persistent gaps. on top of these, medbench @xcite , medbench @xcite , and medgpteval @xcite offer automated, cloud-based evaluation pipelines and blinded expert review to assess chinese medical llms across clinical reasoning, communication, and robustness. ## llmeval-med llmeval-medicine aims to provide a systematic and comprehensive evaluation of medical foundation models. we first detail the dataset, including its construction methodology and data taxonomy in section 3.1. we then outline the evaluation framework and the metrics employed in section 3.3 . ## conclusion we present llmeval-med, a benchmark focused on generative medical tasks across five core dimensions. we open-source the dataset and expert-designed checklists, achieving high humanmachine agreement for most evaluation tasks. our results show that while llms perform well in knowledge recall, they struggle with reasoning and text generation. meanwhile, for fully open-ended medical text generation, human evaluation remains indispensable due to the exceptionally high standards required. our work provides a more accurate and reliable framework for assessing medical llms.