K now L ab_ AIM ed at MEDIQA - CORR 2024: Chain-of-Though ( C o T ) prompting strategies for medical error detection and correction (2024)

Abstract: This paper describes our submission to the MEDIQA-CORR 2024 shared task for automatically detecting and correcting medical errors in clinical notes. We report results for three methods of few-shot In-Context Learning (ICL) augmented with Chain-of-Thought (CoT) and reason prompts using a large language model (LLM). In the first method, we manually analyse a subset of train and validation dataset to infer three CoT prompts by examining error types in the clinical notes. In the second method, we utilise the training dataset to prompt the LLM to deduce reasons about their correctness or incorrectness. The constructed CoTs and reasons are then augmented with ICL examples to solve the tasks of error detection, span identification, and error correction. Finally, we combine the two methods using a rule-based ensemble method. Across the three sub-tasks, our ensemble method achieves a ranking of 3rd for both sub-task 1 and 2, while securing 7th place in sub-task 3 among all submissions.

## introduction the rise of large language models (llms) such as gpt4 @xcite , med-palm @xcite , and llama @xcite have inspired investigations into their potential use in automatically analysing electronic health records (ehrs). however, the usefulness of llms in clinical settings remains challenging due to the fact that these models are trained on large-scale corpora which may contain inaccuracies, common mistakes, and misinformation @xcite . to motivate research on the problem of identifying and correcting common sense medical errors in clinical notes using llms, the mediqa-corr (medical error detection correction) shared tasks are proposed. herein, we describe our submissions to the shared tasks presenting two methodologies and an ensemble approach using gpt4, all utilising in-context learning (icl) @xcite in conjunction with chain-of-thought (cot) @xcite and reason prompts. the ensemble method achieves accuracies of 69.40% and 61.94% for sub-task 1 and sub-task 2, respectively, while obtaining a bluert score of 0.6541 for sub-task 3. ## discussion our cot prompting strategy works well in conjunction with the rag system. as depicted in figure 3 , across various few-shot settings (e.g., 2, 3, 4, and ## conclusion we present our submission to the mediqa-corr shared task for medical error detection and correction. our study evaluates the effectiveness of the gpt4 model through various prompting strategies employing cot prompting and reasoning methods. specifically, our cot prompting strategies achieve high accuracies in error detection and identification tasks. additionally, our ensemble method, which combines outputs from both methods, demonstrates a better performance on the nlg task than the cot prompting alone. in the future, we aim to explore our approach for other downstream tasks in the clinical domain using open-source llms.