EMPEC : A Comprehensive Benchmark for Evaluating Large Language Models Across Diverse Healthcare Professions (2025)

Abstract: Recent advancements in Large Language Models (LLMs) show their potential in accurately answering biomedical questions, yet current healthcare benchmarks primarily assess knowledge mastered by medical doctors, neglecting other essential professions. To address this gap, we introduce the Examinations for Medical PErsonnel in Chinese (EMPEC), a comprehensive healthcare knowledge benchmark featuring 157,803 exam questions across 124 subjects and 20 healthcare professions, including underrepresented roles like Optometrists and Audiologists. Each question is tagged for release time and source authenticity. We evaluated 17 LLMs, including proprietary and open-source models, finding that while models like GPT-4 achieved over 75% accuracy, they struggled with specialized fields and alternative medicine. Notably, we find that most medical-specific LLMs underperform their general-purpose counterparts in EMPEC, and incorporating EMPEC’s data in fine-tuning improves performance. In addition, we tested LLMs on questions released after the completion of their training to examine their ability in unseen queries. We also translated the test set into English and simplified Chinese and analyse the impact on different models. Our findings emphasize the need for broader benchmarks to assess LLM applicability in real-world healthcare, and we will provide the dataset and evaluation toolkit for future research.

## introduction recent advancements in large language models (llms) have demonstrated the potential of llmbased artificial intelligence (ai) in providing accurate answers to questions about world knowledge. these advancements are reflected in a series of studies and models, including but not limited to gpt-4, gemini, mistral, and llama series @xcite @xcite . to benchmark the internal knowledge of llms, multiple datasets @xcite @xcite have been introduced, focusing on their ability to measure and respond with accurate information. more recently, there has been a significant push towards adapting llms for use in the biomedicine domain @xcite @xcite , exploring the possibility of deploying these models in real healthcare scenarios. to assess the effectiveness of llms in healthcare, considerable effort @xcite @xcite @xcite has been invested in benchmarking their capabilities. among these efforts, several studies @xcite have incorporated long-form diagnostic questions into their benchmarks to more evaluate the models' capacities to function akin to real physicians. in another line of work, multiple-choice questions have emerged as a straightforward and objective means of evaluation @xcite @xcite . these studies mostly utilize questions from medical licensing exams, research papers, and textbooks to assess llms' knowledge and suitability to serve in a physician-like role. existing medical benchmarks are limited in scope and authenticity in real-world healthcare environments, medical doctors, while integral, represent only a fraction of the entire healthcare system. the international standard classification of occupations (isco) by the international labour organization categorises health professionals into two sub-major groups, with medical doctors listed alongside ten other minor occupational groups. these include nursing and midwifery professionals and paramedical practitioners, among others. each minor group comprises at least two unit groups, which in turn consist of multiple professions. this diversity in healthcare roles highlights that the expertise of medical doctors is just a subset of the broader healthcare knowledge spectrum. nevertheless, as shown in in response to the identified research gap, we introduce empec, a comprehensive large-scale healthcare knowledge benchmark. empec comprises 157,803 exam questions spanning 124 subjects across 20 healthcare professions. this comprehensive benchmark goes beyond the commonly assessed professions such as physicians and nurses, to include occupations like optometrist and audiologist often overlooked in previous assessments, thereby filling a critical void in existing benchmarks. the benchmark is originally in traditional chinese; to broaden the applicability of our work beyond chinese-speaking regions, we add an english translation of the test set generated by gpt-4. empec stands out not only for its substantial size and extensive coverage but also for its authoritative and time-sensitive features. each question within empec is tagged with its release time, ensuring that the benchmark continuously integrates the latest questions from the source as soon as they are made available. this feature effectively mitigates the risk of data contamination @xcite during evaluation processes, thereby maintaining the benchmark's relevance and accuracy over time. furthermore, the provenance of every question in empec is documented, with each item linked to its original release. this verification ensures the benchmark's reliability and authenticity, positioning it as a critical tool for the evaluation of llms within the healthcare sector. we conducted extensive experiments on 17 of works focussing on benchmarking knowledge within the models. mmlu @xcite and arc @xcite collected various questions from a diverse set of subjects aiming to comprehensively assess the world knowledge of models. gsm8k @xcite collected grade school math word problems to benchmark models' mathematics reasoning ability. finben (xie et al., 2024b) aggregated 35 datasets across 23 financial tasks aiming to fully assess llms' knowledge and capacity in finance. lawbench (fei et al., 2023) compiled datasets to evaluate llm's legal capacities from three cognitive levels. however, in healthcare, most existing benchmarks concentrate on examining physicians' expertise, being unable to assess models' knowledge in the whole healthcare system. ## analysis in general, all the tested models show unsatisfying performance. the average accuracy of 12 out of the 17 tested models is under 60% on empec where random guessing can reach about 25%. the results suggest the difficulty of our benchmark and the healthcare knowledge gap in existing llms. ## conclusion and discussions we introduced empec, the most comprehensive healthcare knowledge benchmark to date, encompassing 157,803 questions across 124 subjects and 20 healthcare occupations. empec goes beyond the physician-centric focus of previous medical benchmarks to provide a holistic assessment of the knowledge of llm in a wide spectrum of health disciplines. our extensive experiments, which tested both proprietary and open-source models, revealed several important findings. although the best general llms performed reasonably well in common professions such as physicians and nurses, they struggled with rarer specialities like dental technicians, optometrists, and tcm practitioners. this highlights the need for more work to improve the knowledge of llms in healthcare beyond the expertise of physicians that has been the primary focus so far. interestingly, we found that existing medical domain-specific llms did not perform better on empec compared to their general counterparts not specialised in healthcare. furthermore, our experiments indicate that the models' performance on the empec test set can predict their effectiveness in addressing unseen healthcarerelated queries. we also found that the transition from traditional to simplified chinese characters limitations empec has several limitations: some professions included in empec have a relatively smaller representation compared to others, which may restrict empec's effectiveness in evaluating knowledge in those specific professions. the english translation is generated by llms without further examination of experts. societal impacts although empec is designed to improve the evaluation of llm in the medical field, it should not be used to assess individual medical competence or to diagnose any patient. any conclusions drawn from models trained on this dataset should take into account its limitations. the dataset's use should be confined to research purposes to prevent potential misuse. in addition, as the simplified chinese version and english version are obtained including conversion and translation, we cannot guarantee the accuracy are fully aligned with the original questions. the datasets are built using publicly available open-source data from taiwan's ministry of examination. their use is encouraged, provided the source is clearly cited. furthermore, the questions in the datasets do not contain any private information. # filter ops -average_line_length_filter: # filter text with the average length of lines max_len: 1500 -language_id_score_filter: