Overview of the MEDIQA - CORR 2024 Shared Task on Medical Error Detection and Correction (2024)

Abstract: Automatic detection and correction of medical errors enables a more rigorous validation of medical documentation as well as clinical notes generated by large language models. Such solutions can ensure the accuracy and medical coherence of clinical texts and enhance patient care and health outcomes. The MEDIQA-CORR 2024 shared task focused on detecting and correcting different types of medical errors in clinical texts. Seventeen teams participated in the shared task and experimented with a broad range of approaches and models. In this paper, we describe the MEDIQA-CORR task, datasets, and the participants’ results and methods.

## introduction a recent survey study from three us health care organizations showed that 1 in 5 patients who read a clinical note reported finding a mistake and 40% perceived the mistake as serious. among the very serious errors reported by patients, the most common category of mistakes was related to current or past diagnoses. other very serious patient-reported mistakes included inaccurate description of medical history, medications or allergies, physical examination, test results, notes on the wrong patient, and sidedness (left vs right) @xcite . @xcite focused on diagnostic errors and analyzed patient-and family-reported error narratives to explore factors that contribute to diagnostic errors. problems related to patient-physician interactions emerged as major contributors. the probability of such errors is expected to increase in medical documents and clinical notes generated by large language models (llms) to assist healthcare professionals in their daily documentation tasks. on a general, coarse-grained, level, llms have shown the ability to imitate clinical reasoning while forming mostly accurate diagnoses @xcite . however, one of the main challenges in integrating llms in medical workloads is their potential to generate misleading or incorrect information @xcite . rigorous validation processes are essential to mitigate these risks and make llms safe(r) to use for medical content generation @xcite . one important aspect of this validation is medical common-sense checking to validate the coherence and soundness of the generated medical reasoning. however, most previous studies on common sense detection have focused on the general domain @xcite . in this task, we tackle the problem of identifying and correcting (common sense) medical errors in clinical notes. from a human perspective, identifying and correcting these errors requires medical expertise, specialized knowledge, and sometimes practical experience. to the best of our knowledge, this task is the first to address the automatic validation and correction of clinical notes. ## task description the mediqa-corr 2024 shared task foot_0 addresses the problem of identifying and correcting (common sense) medical errors in clinical notes. from a human perspective, identifying and correcting these errors require medical expertise and knowledge. in the task data, each clinical text is either correct or contains one error. the task consists of three subtasks: a: predicting the error flag (1: the text contains an error, 0: the text has no errors) b: extracting the sentence that contains the error for flagged texts (-1: the text contains no error; sentence id: if the text contains an error) c: generating a corrected sentence for flagged texts ## data creation we created a new dataset of 3,848 clinical texts with injected errors such as diagnosis, causal organism, management, treatment, and pharmacotherapy @xcite . the dataset includes two types of texts: clinical texts from publicly available data (ms collection) and de-identified clinical notes from the university of washington medical center (uw collection). the uw dataset was built using new de-identified notes and requires signing a data usage agreement. the ms dataset was built by transforming the medqa medical questionanswering dataset @xcite with manual error injections and text modifications that leveraged the clinical notes and the multiple-choice questions. the ms training set contains 2,189 clinical texts. figure 1 presents examples from the ms training data. the ms validation set contains 574 clinical texts and the uw validation set contains 160 clini-cal texts. the final test set consists of 597 clinical texts from the ms collection and 328 clinical texts from the uw dataset. ## conclusion the mediqa-corr shared task was tackled by a wide variety of approaches from the participating teams. ranging from algorithmic reasoning approaches leveraging the llms as intermediate extraction tools (e.g., for ner) to approaches that are fully controlled by llms and prompting techniques. the best performing methods were datasetdependent, i.e., different methods or parameters were used for each dataset. generalized, datasetagnostic, approaches fared reasonably well in comparison. a key challenge was in detecting correctly which text and which sentence contained errors, with only two teams reaching an accuracy above 70% in text flagging, and only one team reaching an accuracy greater than 65% in detecting the sentence containing the error. the detection accuracy impacted the quality of the corrected texts (e.g., providing corrections when the sentence contained no errors) but the correction results were less contrasted in general with six teams reaching an aggregate score greater than 70%. moving forward, optimizing the dataset-agnostic approaches is likely to be a key focus as it has the most impact on production-grade models/systems for clinical note generation/validation. the data provided by mediqa-corr can be leveraged for that as they showed to be sufficiently challenging to be used as a benchmark for generalized approaches.