W ang L ab at MEDIQA - CORR 2024: Optimized LLM -based Programs for Medical Error Detection and Correction (2024)

Abstract: Medical errors in clinical text pose significant risks to patient safety. The MEDIQA-CORR 2024 shared task focuses on detecting and correcting these errors across three subtasks: identifying the presence of an error, extracting the erroneous sentence, and generating a corrected sentence. In this paper, we present our approach that achieved top performance in all three subtasks. For the MS dataset, which contains subtle errors, we developed a retrieval-based system leveraging external medical question-answering datasets. For the UW dataset, reflecting more realistic clinical notes, we created a pipeline of modules to detect, localize, and correct errors. Both approaches utilized the DSPy framework for optimizing prompts and few-shot examples in large language model (LLM) based programs. Our results demonstrate the effectiveness of LLM based programs for medical error correction. However, our approach has limitations in addressing the full diversity of potential errors in medical documentation. We discuss the implications of our work and highlight future research directions to advance the robustness and applicability of medical error detection and correction systems.

## introduction medical errors pose a significant threat to patient safety and can have severe consequences, including increased morbidity, mortality, and healthcare costs. detecting and correcting these errors in clinical text is crucial for ensuring accurate medical documentation and facilitating effective communication among healthcare professionals. one of the fastest-growing use cases for artificial intelligence (ai) in healthcare is clinical note generation, often from transcriptions of physician-patient dialogues. however, assessing the quality and accuracy of these notes is challenging, and automated detection and correction of errors could have a significant impact on patient care. the reliability of large language models (llms) in critical applications, such as healthcare, is a major concern due to the potential for hallucinations (generating false or nonsensical information) and inconsistencies. robust solutions to the question of error detection and correction are essential for addressing these concerns and enabling the safe and effective use of llms in medical contexts. the mediqa-corr 2024 (ben abacha et al., 2024a) shared task focuses on identifying and correcting medical errors in clinical notes. each text is either correct or contains a single error. the task involves three subtasks: (1) detecting the presence of an error, (2) extracting the erroneous sentence, and (3) generating a corrected sentence for flagged texts. in this paper, we present our approach, which achieved the top performance across all three subtasks in the mediqa-corr 2024 competition. we develop a series of llm-based programs using dspy, a framework for optimizing prompts and few-shot examples. we provide a detailed description of our methodology and results, followed by a discussion of the implications of our work and future directions in the field of medical error detection and correction. ## related work the use of large language models (llms) in medicine has attracted considerable attention in recent years. the release of llms such as gpt-4 has led to intensive research in the medical community @xcite , particularly in clinical note generation. the mediqa-chat 2023 @xcite competition showcased the performance of automated note generation solutions @xcite , and further work has demonstrated that llms can sometimes outperform humans on clinical text summarization tasks @xcite . however, there has been limited research focusing on granular audits of these clinical notes with respect to accuracy and error correction. the mediqa-corr 2024 shared task addresses this gap by providing a platform for researchers to develop and evaluate novel approaches to error detection and correction in clinical text, ultimately contributing to the development of more reliable ai systems in healthcare. ## task description the mediqa-corr 2024 shared task provides two distinct datasets: ms and uw @xcite . the ms dataset consists of a training set containing 2,189 clinical texts and a validation set (#1) containing 574 clinical texts. the uw dataset, on the other hand, consists solely of a validation set (#2) containing 160 clinical texts. the test set for the shared task includes clinical texts from both the ms and uw collections. the evaluation metrics for the mediqa-corr 2024 shared task vary across the three subtasks: the composite score for each text in subtask 3 is calculated as follows: ## conclusion the approach presented in this paper, which combines retrieval-based methods, few-shot learning, and systematic prompt optimization, demonstrates the potential of ai-assisted tools for detecting and correcting medical errors in clinical text. the strong performance achieved across all three subtasks of the mediqa-corr 2024 shared task highlights the effectiveness of our methods in addressing the specific challenges posed by different datasets and error types. however, further research is necessary to extend the applicability of our approach to a wider range of medical contexts, incorporate domain-specific knowledge, and integrate with existing clinical systems. as the field of ai-assisted medical error detection and correction continues to evolve, collaboration between ai researchers and healthcare professionals will be crucial to develop solutions that effectively augment and support clinical decision-making processes, ultimately contributing to improved patient safety and healthcare quality.