C li M ed B ench: A Large-Scale C hinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios (2024)

Abstract: With the proliferation of Large Language Models (LLMs) in diverse domains, there is a particular need for unified evaluation standards in clinical medical scenarios, where models need to be examined very thoroughly. We present CliMedBench, a comprehensive benchmark with 14 expert-guided core clinical scenarios specifically designed to assess the medical ability of LLMs across 7 pivot dimensions. It comprises 33,735 questions derived from real-world medical reports of top-tier tertiary hospitals and authentic examination exercises. The reliability of this benchmark has been confirmed in several ways. Subsequent experiments with existing LLMs have led to the following findings: (i) Chinese medical LLMs underperform on this benchmark, especially where medical reasoning and factual consistency are vital, underscoring the need for advances in clinical knowledge and diagnostic accuracy. (ii) Several general-domain LLMs demonstrate substantial potential in medical clinics, while the limited input capacity of many medical LLMs hinders their practical use. These findings reveal both the strengths and limitations of LLMs in clinical scenarios and offer critical insights for medical research.

## introduction with the advent of chinese medical large language models (llms) such as huatuogpt @xcite , chatmed (zhu and wang, 2023), and bentsao @xcite , the potential for these tools in healthcare has expanded considerably @xcite . these models are engineered to address intricate medical problems by providing diagnostic assistance and treatment suggestions. nonetheless, the absence of a comprehensive and systematic evaluation of their performance, which encompasses aspects such as re-sponse accuracy, hallucination incidence, and content safety, hampers their integration into clinical practice. consequently, there is an urgent need for a standardized evaluation benchmark to scrutinize the capabilities and limitations of such medical llms. developing a practically relevant benchmark is non-trivial. there is a substantial disconnect between current benchmarks for the chinese language and the realities of medical practice, as such benchmarks are mostly derived from open educational resources @xcite . a benchmark based on real-world medical cases offers superior authenticity and heterogeneity, while more accurately mirroring the intricacies encountered in clinical practice. these cases present greater challenge and complexity, leading to a more rigorous assessment of model performance and robustness in practical applications, including clinical decision support, diagnosis, and treatment recommendations. moreover, benchmarks developed from open resources are susceptible to data contamination issues. prevalent medical benchmarks like medqa @xcite and medmcqa @xcite incorporate data from accessible sources such as textbooks, scholarly articles, and qualification examinations. the effectiveness of such evaluation benchmarks is controversial: medical exams are inefficient clinical performance indicators. large-scale ehr-based benchmarks such as emr-qa @xcite have addressed the deficiency in clinical qa; however, language discrepancies preclude their direct applicability for evaluating chinese medical llms. chinese benchmarks, including cmexam @xcite , cmb @xcite , and mlec-qa @xcite , primarily source their data from exams such as cnmle and nmlec. medbench @xcite uses exam questions and artificially generated ehrs to evaluate the llms' exam-solving capabilities in different subdisciplines. despite their comprehensive analysis, these benchmarks are disconnected from actual medical practice due to their lack of real-world medical case data, and the use of exam-based data raises concerns about data pollution. the need for good benchmarks makes evaluating performance a significant challenge. to address the limitations of prior research, we introduce the climedbench, a robust benchmark comprising 33,735 questions across 14 core medical scenarios assessing llms' ability across six dimensions, primarily sourced from authentic cases to align with medical standards and practices. the climedbench integrates expertise from chinese medical practitioners, offering a valid measure for gauging medical linguistic proficiency and cognitive skills in llms. we evaluate various general and medical-specific llms using this benchmark and perform a comprehensive analysis that sheds light on relevant research avenues to enhance the medical capabilities of llms. the main findings on this benchmark are as follows: ## models and evaluation metrics to assess the state-of-the-art, we conduct evaluations using climedbench of 11 representative llms from both the general and medical domains, including openai's gpt series foot_0 (gpt-3.5-turbo-1106 and gpt-4-1106-preview), chatglm3 ## main results we conduct an in-depth evaluation of 11 llms using climedbench, stringently examining their performance across seven pivot dimensions. corresponding comparisons utilizing automatic metrics are provided in several chinese llms (ernie-bot and qwen) demonstrate performance on par with gpt-4 in clinical medicine of china, achieving scores primarily within the range of 68.5 to 69.2. this could stem from the unique treatments, expression styles, and china-manufactured pharmaceuticals, which diverge from what is encountered in the training data of gpt series models. a disparity of capabilities between these chinese llms and the gpt series predominantly manifests in medical knowledge and reasoning. next, we will summarize the performance of llms with regard to particular evaluation dimensions. regarding clinical qa abilities, qwen outperforms others with an average score of 88.7. however, variability in model performance across scenarios is evident, with chatgpt achieving the highest score (97.4) on id #2 but not ranking among the top performers in other scenarios. gpt4 and ernie-bot show exceptional reasoning capabilities, achieving average scores of 75.1 and 73.3, respectively. the notable performance disparity between general and medical-specific llms high-lights the need for further enhancement in the reasoning ability of medical-specific llms. in all evaluated models, hallucinations are significantly pronounced. the fit data is designed to trigger hallucinations by incorporating an erroneous reference. their data sources are the same as bkt, however, model accuracy exhibits a marked reduction, plummeting from an average score of 47.3 to 8.3. this substantial decline shows the vulnerability of llms to uncritically adopt perspectives presented in their input, highlighting an immediate need for enhancement. hallucinations exhibited on the wtd dataset indicate that for questions with special structures, llms not only need to master the knowledge points examined by the questions but also need to understand the logical relationships in the questions, which may exceed the ability of the models. the knowledge application ability of the leading general llms ranges from 79.8 to 82.4, suggesting a substantial reservoir of medical knowledge of these models. for the information retrieval task, medicalgpt has significantly narrowed the disparity with leading llms, achieving a score of 33.3, merely 8.7 points below the topperforming ernie-bot. this improvement predominantly stems from the specialized nature of its generated terminologies. ## quantitive analysis chain-of-thought to demonstrate the potential improvement of llm's reasoning abilities with customized prompts, we compare the performance of four representative models using vanilla and chain-of-thought (cot) prompts in figure 5 . we observe that the utilization of tailored cot prompts significantly enhances model performance across seven datasets that demand higher reasoning skills. specifically, for qwen, there is an average accuracy increase from 62.7% to 69.2%. it suggests that cot can enhance reasoning and hallucination resistance in medical contexts, as observed in the surgical organization, false info test, and wrong treatment datasets. conversely, the impact of cot prompts on chatglm3 is minimal and it adversely affects huatuogpt, underscoring the dependency of cot prompt efficacy on the model's comprehension and contextual correlation proficiency. in addition, the long text of the few-shot cot prompt (on average 4.987 times longer than the vanilla prompt) is also a reason for the decrease in accuracy, as described in the following paragraph. ## benchmark validity verification to ensure the real-world applicability and validity of climedbench in clinical settings, the design rationale of the 14 core scenarios is meticulously aligned with actual clinical practices. for instance, the in-hospital diagnosis datasets are carefully structured to adhere to established clinical pathways in healthcare settings. to further confirm the reliability of climedbench, we engaged medical professionals to assess our benchmark from three perspectives, including medical accuracy, assessment effectiveness, and language proficiency. figure 9 plots the assessment results, which substantiate the quality of climedbench with an "acceptable" (3 points) or higher rating. furthermore, we calculate the spearman correlation between our climedbench and another representative benchmark, namely medbench foot_7 based on other kinds of data. this approach foot_8 allows us to conduct multidimensional evaluations that reflect both collective and discrete correlations between benchmarks. figure 9 illustrates a robust correlation between the climedbench and medbench leaderboards, with an overall spearman correlation of 0.943 and subdivisions no less than 0.657, substantiating climedbench's utility and reliability as an evaluative benchmark. question output debate reflection question set mps . . . . . . item response theory cat item response theory selection algorithm llm test llm ability questions difficulty question pool best-fitting selection estimation result llm a llm b llm a figure 10: the workflow of agent-based cat. theoretical basis our approach is fundamentally rooted in the psychometric item response theory (irt). we incorporate the three-parameter logistic model (irt-3pl), formulated as: procedure of agent-based cat as depicted in figure 10 , our agent-based cat consists of two main steps: multi-agent based participant synthesis (mps) and computerized adaptive testing (cat). mps leverages multi-agent llms to synthesize data that mimics participant behavior, which is used to construct a question pool. we then use cat to sequentially select the best-fitting questions from the question pool to evaluate the ability of llms. algorithm 1 provides the "generation-debatereflection" process in mps for data synthesis, aiming to overcome the difficulty of insufficient participant data in previous cat @xcite . once a sufficient number of participant behavior data is synthesized, the modeling process aligns more closely with the assumptions of irt. specifically, we use permutations of 5 llms (e.g., bloomz and chatglm2) to form multi-agent based participants, serving as the examinees in the irt process to synthesize performance-related data. ## related work multimedqa @xcite and pub-medqa @xcite are effective benchmarks based on qa tasks for evaluating the medical abilities of llms. large-scale ehr-based benchmarks such as emr-qa @xcite have addressed the deficiency in clinical qa, but language discrepancies preclude their direct applicability for evaluating chinese medical llms. the assessment of llms for chinese medical proficiency has traditionally relied on benchmarks derived from multiple-choice and generative question-answering formats, utilizing resources like exam questions, textbooks, and doctor-patient interactions. chinese benchmarks, including cmexam @xcite , cmb @xcite , mlec-qa @xcite , and medqa @xcite , primarily source their data from exams such as cnmle and nmlec. despite their comprehensive analysis, these benchmarks are disconnected from actual medical practice due to their lack of real-world medical case data, and the challenge of ensuring quality control and avoiding data pollution grows proportionally with the involved volume of data @xcite . @xcite collected more than 21k multi-choice questions from the national licensed pharmacist examination in china, but only the test set of the dataset has been released. finally, we contrast our work with other lines of work sharing seemingly similar goals. med-bench @xcite ) is an exhaustive benchmark designed for the domain of chinese medical qa, it utilizes exam questions and synthetical ehrs to evaluate the llms' exam-solving capabilities in different areas, rather than actual clinical skills. in contrast, our benchmark extends this framework across 14 diverse medical scenarios. furthermore, despite medbench providing preliminary empirical analysis, it lacks in-depth qualitative analyses of the model's performance. rjua-qa @xcite creates high-quality medical datasets to evaluate clinical reasoning based on ehrs and clinical cases. however, it is restricted to urology, offering limited insight into the broader medical capabilities of llms. ## conclusion and discussions this paper introduces climedbench, a robust benchmark derived from real medical cases that comprises 33,735 questions across 14 core medical scenarios assessing llms' ability across six dimensions. evaluating diverse llms reveals their suboptimal performance, especially where medical reasoning and factual consistency are vital, underscoring the need for advances in clinical knowledge and diagnostic accuracy. we also conducted a comprehensive qualitative analysis of the experimental outcomes and made several novel insights. simultaneously, we proposed the agent-based cat approach, which enables rapid assessment with minimal problem sets.