M ed E val: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation (2023)

Abstract: Curated datasets for healthcare are often limited due to the need of human annotations from experts. In this paper, we present MedEval, a multi-level, multi-task, and multi-domain medical benchmark to facilitate the development of language models for healthcare. MedEval is comprehensive and consists of data from several healthcare systems and spans 35 human body regions from 8 examination modalities. With 22,779 collected sentences and 21,228 reports, we provide expert annotations at multiple levels, offering a granular potential usage of the data and supporting a wide range of tasks. Moreover, we systematically evaluated 10 generic and domain-specific language models under zero-shot and finetuning settings, from domain-adapted baselines in healthcare to general-purposed state-of-the-art large language models (e.g., ChatGPT). Our evaluations reveal varying effectiveness of the two categories of language models across different tasks, from which we notice the importance of instruction tuning for few-shot usage of large language models. Our investigation paves the way toward benchmarking language models for healthcare and provides valuable insights into the strengths and limitations of adopting large language models in medical domains, informing their practical applications and future advancements.

## introduction recent advanced language models, e.g., gpt-3, chatgpt, and llama @xcite , are effective in various general tasks, suggesting their potential to healthcare use cases, such as alleviating the burden on human experts in decisionmaking and patient care. however, training, adapt1 we will release the data set and source codes at https: //github.com/zexuehe/medeval the data set will also be available at the department of veterans affairs open data portal https://www.data.va.gov ing, and evaluating these models requires highquality domain-specific datasets, which are often challenging to obtain. previous medical datasets have been collected from healthcare-related literature @xcite @xcite or web pages on the internet @xcite . while these datasets are large, they may lack quality with heterogeneous topics (e.g., scientific literature about nutrition may offer limited help in the decision-making process of an x-ray analysis). on the other hand, high-quality clinical data is typically obtained by annotating records from healthcare systems like mimic-cxr @xcite . however, such data is either limited in size @xcite , or may only cover certain dominant systems and specific domains foot_0 , such as chest x-rays @xcite or eye diseases @xcite . other approaches involve automatically generating medical corpus using templates @xcite or using language models @xcite ), but they have been noted to be limited in diversity, complexity, and quality @xcite . to tackle the aforementioned challenges and facilitate research in clinical nlp, we introduce medeval, a large-scale medical benchmark with multi-level curated labels for multiple tasks and multiple domains. medeval comprises 22,779 sentence-level datapoints from radiology reports, including expert-crafted classification labels (e.g., abnormality identification labels) and ground truth for generation tasks (e.g., disambiguated rewritings). additionally, we include 21,228 complete reports with expert-annotated medical codes for disease classification (e.g., for ankle radiology studies) and golden output for generation tasks (e.g., summarization of radiology reports). besides the ability to support multi-tasks at different levels, mede-val's uniqueness also lies in its diverse data coverage for different body parts (such as chest, foot, and ankle) and different modalities (x-rays, ultrasound, etc.), and the incorporated novel tasks/data that are collected from the u.s. department of veterans affairs (va) health care system nationwide. to the best of our knowledge, medeval represents the first expert-curated medical nlp benchmark that is both comprehensive and large-scale. mede-val will be released to facilitate future research. we further conduct a comprehensive evaluation of multiple state-of-the-art language model baselines, including domain-adapted plms followed by in-domain fine-tuning (e.g., fine-tuned bert @xcite ) and general-purposed llms utilized with few-shot in-context learning (e.g., chat-gpt). we evaluate their performance on sentencelevel and document-level nlu and nlg tasks. we observe the effectiveness of both categories of models in different healthcare tasks, with surprisingly comparable performances from llms only using few-shot learning to domain-adapted plms in certain generation tasks. our comprehensive evaluation indicates language models are strong candidates in medical tasks whose data is already seen/similar to their training data. our investigation provides insights into the potentials and x-ray, ct, ultrasound, etc) and examined body parts (e.g., chest, abdomen, etc). limitations of llms in healthcare domains, guiding the appropriate use of llm-assisted healthcare decision-making systems in the future. overall, our contributions are summarized as: medical benchmark existing medical benchmarks are typically collected from the following resources. first, data is crawled from public resources, such as biomedical journals and literature like pubmed central (pmc) or semantic scholar ammar et al. (2018), or healthcare-related web pages such as mqp (mccreery et al., 2020) which was proposed by collecting covid-19 faqs from the internet. those data are heterogeneous and may lack relevance for assisting certain clinical purpose. on the other hand, high-quality benchmarks are extensively collected from real-world healthcare systems, such as the medical information mart for intensive care databases (mimic (johnson @xcite ). several works have been proposed based on their databases. for instance, mimic-cxr @xcite is a dataset consisting of pairs of radiology images and reports of chest x-ray exams. mimic perform dataset @xcite comprises physiological signals related to critically-ill patients. additionally, @xcite collected document summary pairs labeled with diagnosis and procedure codes from @xcite . though their collection may be one of the most comprehensive, there are many domains not included. a more recent attempt was made to alleviate the incompleteness concern by introducing m3 @xcite , a multi-domain medical benchmark that incorporates multi-level expert annotations. by only considering studies on ophthalmology, m3 offers limited help in providing a fully comprehensive solution. to complement these existing efforts, we collect a large-scale dataset of real medical reports from another healthcare system that offers broader coverage including 35 human body regions from 8 examination modalities (e.g., x-ray, ct, etc.). language models for healthcare large pretrained language models are being widely adopted to solve healthcare tasks. one line of research involves adapting general language models to the biomedical domain through continuous training on domain-specific data and tasks. for instance, @xcite enhanced bert with contrastive learning for chest report generation. clinicbert @xcite was proposed by continuously training bert on clinic notes using masked language modeling, and @xcite developed radbert by continuously training bert on a vast collection of radiology reports. other adaptations of bert, such as biobert @xcite , blue-bert @xcite , scibert @xcite , and biomegatron @xcite , involved training on large publicly available medical corpora like pubmed or semantic scholar. furthermore, llms of alternative architectures have also been employed, including bioelmo @xcite , biobart @xcite , and biomed-roberta @xcite . another research direction capitalizes on the generaliza-tion capabilities of recent llms, where biomedical problems are addressed through prompting llms in zero-shot or few-shot settings. this approach has been utilized in various applications, such as medical report summarization @xcite , medical writing @xcite , and medical named entity recognition @xcite , etc. in this work, we propose medeval, a multilevel data with curated annotations at various granularity to comprehensively evaluate the strengths and limitations of lms in healthcare. for each evaluation task, we split the data in a ratio of 7:1:2 for train/validate/test. ## results and discussion in this section, we first present the results for sentence-level nlu tasks (ambiguity identification and abnormality identification) in the effectiveness of instruction tuning while biomed lm is the first large language model customized for the biomedical domain, we observe that it does not outperform adapted plms and most prompted llms in the majority of tasks. particu bold: the highest performance. underlined: the lowest. in zero-shot nlu tasks, only 40% of the test cases receive appropriate responses at the sentence level and the qualified rate drops to less than 1% at the document level (so we did not report the results in in the remainder of this section, we focus on addressing more intriguing questions based on average performance across a range of baselines (e.g., the average accuracy of adapted plms versus prompted llms), where we exclude biomed lm from further consideration. discussion on task type and granularity in this section, we aim to determine the proficiency of language models at different levels and tasks. to achieve this, we begin by calculating the average accuracy scores of all adapted plm baselines and first, examining the results presented in figure 3 , we observe that both adapted plms and prompted llms perform relatively similarly across different data levels. however, it becomes apparent that adapted plms outperform prompted llms in nlu tasks, no matter whether it's on the sentence or document level. this suggests that fine-tuning provides a more effective means of injecting specific knowledge about narrow domains or tasks. on the other hand, consistently superior performance of prompted llms compared to adapted plms is observed in generation tasks, at both the sentence and document levels. this can be attributed to multiple advantages of large-scale pre-training such as a larger model size or the benefits hfrl in the llms we utilized, such as chatgpt. these models demonstrate a capability to generate language that is more akin to human-like expressions, thereby achieving better generation scores. these imply that fine-tuning plm models can be a viable choice for nlu tasks, while prompting-based llms may be more suitable when healthcare professionals require an ai writer to help their work. common v.s. rare domains in ## conclusion we introduce medeval, a multi-task, multi-level, and multi-domain medical benchmark designed to serve as a comprehensive testbed for advanced language models. through extensive evaluation experiments, we thoroughly analyze the capabilities and limitations of current llms in tackling various medical tasks, such as the effectiveness of instruction tuning and the performance disparities between adapted and prompted lms in nlu and nlg tasks. our findings provide valuable insights and serve as a handbook for future research in utilizing llms to enhance healthcare practices.