Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark (2024)

Abstract: The adoption of large language models (LLMs) to assist clinicians has attracted remarkable attention. Existing works mainly adopt the close-ended question-answering (QA) task with answer options for evaluation. However, many clinical decisions involve answering open-ended questions without pre-set options. To better understand LLMs in the clinic, we construct a benchmark ClinicBench. We first collect eleven existing datasets covering diverse clinical language generation, understanding, and reasoning tasks. Furthermore, we construct six novel datasets and clinical tasks that are complex but common in real-world practice, e.g., open-ended decision-making, long document processing, and emerging drug analysis. We conduct an extensive evaluation of twenty-two LLMs under both zero-shot and few-shot settings. Finally, we invite medical experts to evaluate the clinical usefulness of LLMs

## introduction large language models (llms), such as chatgpt @xcite , are increasingly being recognized for their potential in healthcare to aid clinical decision-making. recently, many efforts have been made to develop medical llms @xcite @xcite . existing research shows that medical llms outperform human experts across a variety of medical tasks. in particular, medprompt @xcite and medpalm-2 @xcite have respectively achieved a competitive accuracy of 90.2 and 86.5 compared to human experts 87.0 @xcite on the united states medical licensing examination (usmle). despite the promising results of existing llms, several issues need to be addressed for better use of llms in assisting clinicians: (i) limited evaluation: most works only focus on evaluating llms in close-ended (exam-style) qa tasks, overlooking their evaluation in other scenarios, such as clinical language understanding and generation @xcite ; (ii) limited task: current works primarily focus on non-clinical machine learning tasks, which cannot adequately evaluate the models' ability to solve complex clinical problems, e.g., health education @xcite , treatment recommendation @xcite , and emerging drug analysis; (iii) limited comparison: most works either provide limited qualitative examples (patel and lam, 2023) or use limited baselines (mainly focusing on chatgpt) for quantitative comparisons @xcite . therefore, existing works fall short of providing a thorough comparative analysis of different llms across diverse clinical scenarios and tasks. in this paper, as shown in figure 1 , we propose the clinicbench, which encompasses eleven downstream tasks (across three different scenarios, i.e., reasoning, generation, and understanding) and seventeen datasets to provide a comprehensive evaluation of llms in the clinic. most recently, several works @xcite have attempted to benchmark llms in healthcare. however, they only adopt the non-clinical machine learning tasks from existing benchmarks (i.e., blue @xcite and blurb @xcite ) for evaluation. as shown in finally, to build medical llms, we notice that existing works @xcite mainly adopt medical consultant dialogues, exam-style qa, and articles as fine-tuning data, which, however, cannot directly or fully represent clinical knowledge. for example, dialogues contain irrelevant and incomplete information, and articles may focus on laboratory results rather than clinical practice. thus, we explore using clinical-standard knowledge bases collected from the clinic to develop a medical llm. we then analyze how different types of fine-tuning data affect medical llms' performance. the evaluation proves the importance of introducing clinicalstandard knowledge bases for fine-tuning and increasing the diversity of fine-tuning data. the main contributions of this work are: in the following sections, we introduce the details of our benchmark in section 3. we report the main results of our benchmark in section 4.2. we discuss the differences in llms' performance on machine learning and clinical tasks in section 4.3. we analyze the effect of few-shot learning of llms in medical tasks in section 4.4. we evaluate the clinical usefulness of llms in section 4.5. finally, we analyze the effect of medical ift data in sections 4.6 and 4.7. ## key findings for clarity, we summarize the main findings from our benchmark as follows: ## clinicbench please refer to our supplementary material and appendix a for more details of our benchmark. i) referral qa: when a patient returns to their gp or is referred to another hospital, this task can help clinicians quickly understand the patient's treatment and medication. we randomly extract 1,000 referral letters from mimic-iv @xcite and use gpt-4 to generate multiple qa pairs about medications and treatments for each letter. after review, correction, and filtering by experts from our university's health department, we finally obtain 1,057 q&a pairs. ii) treatment recommendation: it requires providing all possible appropriate medications for treating the current patient's condition, thus helping the clinician designate a treatment plan. we collect 796 patient-physician conversations, which describe the patient's diseases and symptoms, from chatdoctor @xcite . we further collect the corresponding drug recommendations given by the physicians. during our evaluation, we ask the llms to list all available drugs for the treatment of patients based on their diseases or symptoms. discussion unlike exam-style qa tasks with answer options, the treatment recommendation is an open-ended task where the llms need to rely on their own knowledge to reason and make decisions; both patient education and hospitalization summarization tasks require llms to process patient documents with lengths around 2,000-3,000 words, and thus can evaluate llms' ability to deal with long health documents, which are common in the clinic; the tasks of pharmacology qa and drug interaction for emerging drugs are crucial for supporting the decision-making and management of new drugs, which frequently emerge in real-world clinical practice. ## results in this section, we will show the detailed results, analyses, and experimental findings of different llms in our benchmark. ## conclusions in this paper, we build a benchmark for comprehensively evaluating large language models (llms) in the clinic, clinicbench, which includes clinical language reasoning, generation, and understanding scenarios. our presented benchmark comprises seventeen datasets across five machine learning tasks and six complex clinical tasks. we evaluate twenty-two diverse llms ranging from 7 billion to 70 billion model parameters under both zeroshot and few-shot settings to provide insights into the performance of llms in the clinic. we also assess llms' clinical usefulness, i.e., factuality, completeness, preference, and safety, which are essential for reliable deployment in clinical practice. our results reveal a significant gap between the capabilities of llms and the requirements for clinical application, highlighting the challenges llms encounter in providing optimal support in clinical environments. finally, we further analyze the impact of the types and sizes of fine-tuning data and explore the effectiveness of clinical-standard knowledge bases to develop medical llms.