EM _ M ixers at MEDIQA - CORR 2024: Knowledge-Enhanced Few-Shot In-Context Learning for Medical Error Detection and Correction (2024)

Abstract: This paper describes our submission to MEDIQA-CORR 2024 shared task for automatic identification and correction of medical errors in a given clinical text. We report results from two approaches: the first uses a few-shot in-context learning (ICL) with a Large Language Model (LLM) and the second approach extends the idea by using a knowledge-enhanced few-shot ICL approach. We used Azure OpenAI GPT-4 API as the LLM and Wikipedia as the external knowledge source. We report evaluation metrics (accuracy, ROUGE, BERTScore, BLEURT) across both approaches for validation and test datasets. Of the two approaches implemented, our experimental results show that the knowledge-enhanced few-shot ICL approach with GPT-4 performed better with error flag (subtask A) and error sentence detection (subtask B) with accuracies of 68% and 64%, respectively on the test dataset. These results positioned us fourth in subtask A and second in subtask B, respectively in the shared task.

## introduction an estimated 795,000 americans either become permanently disabled or die each year across various healthcare settings due to misdiagnoses of serious diseases, as reported by @xcite . the key process failures, especially in the emergency department, are errors in diagnostic assessment, test ordering, and test interpretation @xcite . therefore there is a growing interest to assist clinicians in automatic medical error identification, if any, in a clinical note. the mediqa-corr 2024 shared task @xcite , hosted by the 6 th clinical natural language processing workshop at naacl 2024, was proposed to encourage research 1 https://github.com/swati-rajwal/em_mixers_ mediqa-corr-naacl-clinicalnlp-2024 (last accessed: 04/24/2024) in medical error identification and correction in clinical texts. from a human perspective, these errors require medical expertise and knowledge to be both identified and corrected. here we describe our submission to the three sub-tasks: error detection, error sentence identification, and error correction. we explore two approaches; the first uses llm for error detection and correction while the second extends the approach by integrating an additional layer of information retrieval. we selected gpt-4 since it has shown good performance on a variety of medical tasks, according to various recent studies @xcite @xcite . out of the two approaches discussed here and implemented, we observed that the second approach performed better as measured by the evaluation metrics (section 4). the results for error flag (sub-task a) and error sentence detection (sub-task b) by our proposed system (approach 2) ranked fourth and second, respectively, in the shared task. 2 shared task and dataset mediqa-corr 2024 proposed the following three sub-tasks. each sub-task builds upon the previous one, creating a sequential process for detecting, identifying, and correcting errors in medical texts. ## proposed approach figure 2 shows the entire framework and the following is the description of the two approaches to this shared task. we used gpt-4 as the llm @xcite and designed a prompt to call the microsoft azure openai gpt-4 turbo (gpt-4-1106-preview) api 2 . this model has a context window of 128,000 tokens and returns a maximum of 4,096 output tokens. we set the temperature parameter to 0 and top_p to 0.95, respectively. for additional information and access to the code used in our study, please refer to the github repository we have made publicly available. ## discussion across multiple evaluation metrics and datasets, approach 2 consistently outperforms approach 1. this indicates that the addition of external knowledge is potentially leveraging more effective strategies for both error detection and error correction. ## conclusion in this paper, we present our submission to the mediqa-corr shared task for medical error detection and correction. we evaluated two approaches: one with in-context learning (icl) and the other an extension with knowledge-enhanced few-shot icl. based on the evaluation metric results, we conclude that knowledge-enhanced fewshot in-context learning is a promising path toward medical error detection and correction. for future work, we plan to experiment the proposed pipeline with other sources of medical information for comparative analysis.