MEDEC : A Benchmark for Medical Error Detection and Correction in Clinical Notes (2025)

Abstract: Several studies have shown that Large Language Models (LLMs) can answer medical questions correctly, even outperforming the average human score in some medical exams. However, to our knowledge, no study has been conducted to assess the ability of language models to validate existing or generated medical text for correctness and consistency. In this paper, we introduce MEDEC (https://github.com/abachaa/MEDEC), the first publicly available benchmark for medical error detection and correction in clinical notes, covering five types of errors (Diagnosis, Management, Treatment, Pharmacotherapy, and Causal Organism). MEDEC consists of 3,848 clinical texts, including 488 clinical notes from three US hospital systems that were not previously seen by any LLM. The dataset has been used in the MEDIQA-CORR 2024 shared task to evaluate seventeen participating systems. In this paper, we describe the data creation methods and we evaluate recent LLMs (e.g., o1-preview, GPT-4, Claude 3.5 Sonnet, Gemini 2.0 Flash, and DeepSeek-R1) for the tasks of detecting and correcting medical errors requiring both medical knowledge and reasoning capabilities. We also conducted a comparative study where two medical doctors performed the same task on the MEDEC test set. The results showed that MEDEC is a sufficiently challenging benchmark to assess the ability of models to validate existing or generated notes and to correct medical errors. We also found that although recent LLMs have a good performance in error detection and correction, they are still outperformed by medical doctors in these tasks. We discuss the potential factors behind this gap, the insights from our experiments, the limitations of current evaluation metrics, and share potential pointers for future research.

## introduction a survey study from us health care organizations showed that one in five patients who read clinical notes reported finding mistakes and 40% perceived the mistake as serious, with the most common category of mistakes being related to current or past diagnoses @xcite . on the other hand, more and more medical documentation tasks (e.g., clinical note generation) are being supported by llms. in multiple studies, llms have shown the ability to answer accurately questions from medical exams @xcite @xcite and to imitate clinical reasoning in providing diagnoses @xcite . however, one of the main obstacles in adopting llms in medical documentation tasks is their potential to generate hallucinations or incorrect information @xcite and harmful content that might alter clinical decision making @xcite . rigorous validation methods are essential to mitigate these risks and make llms safer to use for medical content generation @xcite . relevant benchmarks are required to assess whether such validation can be fully automated. a key task in this regard is the ability to detect and correct medical errors in clinical texts. most previous studies on (common sense) error detection have focused on the general domain @xcite . in this paper, we tackle the problem of identifying and correcting medical errors in clinical texts. from a human perspective, identifying and correcting these errors requires medical expertise, specialized knowledge, and sometimes practical experience. we introduce a new dataset, medec, and experiment with different recent llms (e.g., claude 3.5 sonnet, o1preview, gemini 2.0 flash, and deepseek-r1). to the best of our knowledge, this is the first publicly available benchmark and study on automatic error detection and correction in clinical notes. 2 related work @xcite introduced a benchmark for consistency evaluation and evaluated pretrained language models (e.g, bert, t5, and gpt-2) on three main categories: semantic, logical, and factual consistency. they found that those language models do not perform well in every test case and have a high level of inconsistency in many cases. @xcite investigated the trustworthiness of more recent language models, chatgpt and gpt-4, regarding semantic consistency and found that while both models appear to show an enhanced language understanding and reasoning ability, they often fail at generating logically consistent predictions. in the medical domain, several recent studies evaluated large language model accuracy and consistency. @xcite conducted a study to assess the accuracy and reliability of medical responses generated by chatgpt. thirty-three physicians across 17 specialties generated 284 medical questions with different levels of difficulty and graded chatgpt's answers for accuracy and completeness. while most of the generated text was evaluated by physicians as accurate, there were potential limitations in handling complex medical questions. in chen et al. ( 2024 ) assessed the effect and safety of llm-assisted patient messaging, as one of the earliest applications of llms in electronic health records (ehrs). the fact that llm-assisted responses were more similar to the llm drafts than to the manual responses, together with the improved interphysician agreement, suggested that doctors might adopt the llm's responses and assessments. the study also found that a minority of llm drafts, if left unedited, could lead to severe harm or death. the safe introduction and use of llms in medical documentation tasks requires reliable and automatic validation methods. however, as far as we know, no benchmark was made publicly available to assess the ability of llms in validating existing or generated medical text for correctness and consistency. in this paper, we present medec, the first benchmark for medical error detection and correction in clinical notes. we describe the data creation methods and we evaluate recent state-of-the-art open domain llms for these tasks. the medec dataset has been used in the first shared task on medical error detection and correction, mediqa-corr 2024, to evaluate models and solutions from seventeen participating teams @xcite . ## medec dataset medec contains 3,848 clinical texts from different specialties. eight medical annotators participated in the annotation task. the dataset covers five types of errors: each clinical text in the dataset is either correct or contains one error introduced using one of two different methods: ms (described in section 3.1) and uw (described in section 3.2). the main motivation for using two different data creation methods was to diversify the errors through different error injection approaches (i.e., leveraging questions and answers from medical board exams in ms vs. manual modification of medical entities or spans in original clinical notes in uw). by using varied clinical texts and multiple error injection methods, we aim to enable a more comprehensive evaluation of the models' ability to handle a broader range of scenarios. the medec dataset is available at: https:// github.com/abachaa/medec . the ms subset is publicly available. the uw subset requires signing a data usage agreement (dua). figure 1 presents examples from the medec-ms collection. ## medical error detection & correction approaches in order to evaluate models on medical error detection and correction, we divide the process into three subtasks: if the text has no error; generated sentence/correction: if the text has an error). for comparison, we build llm-based solutions using two different prompts to generate the outputs required to assess the models on the three subtasks: ## conclusion this paper presented the medec benchmark for medical error detection and correction in clinical notes. an empirical evaluation of llm-based methods showed that, while recent llms have a good performance, they are still outperformed by medical doctors. the results of the doctors' annotation showed that the medec dataset is a sufficiently challenging benchmark to assess the ability of models to validate existing or generated notes and to correct medical errors. we hope that this dataset will enable further studies on medical error 22546 medec dataset given the clinical presentation (young male patient with recurrent bacterial infections), family history (maternal uncles affected, suggesting x-linked inheritance), and laboratory findings...