Maven at MEDIQA - CORR 2024: Leveraging RAG and Medical LLM for Error Detection and Correction in Medical Notes (2024)

Abstract: Addressing the critical challenge of identifying and rectifying medical errors in clinical notes, we present a novel approach tailored for the MEDIQA-CORR task @ NAACL-ClinicalNLP 2024, which comprises three subtasks: binary classification, span identification, and natural language generation for error detection and correction. Binary classification involves detecting whether the text contains a medical error; span identification entails identifying the text span associated with any detected error; and natural language generation focuses on providing a free text correction if a medical error exists. Our proposed architecture leverages Named Entity Recognition (NER) for identifying disease-related terms, Retrieval-Augmented Generation (RAG) for contextual understanding from external datasets, and a quantized and fine-tuned Palmyra model for error correction. Our model achieved a global rank of 5 with an aggregate score of 0.73298, calculated as the mean of ROUGE-1-F, BERTScore, and BLEURT scores.

## introduction clinical notes typically include details about the patient's medical history, symptoms, physical examinations, diagnostic tests, treatments administered, and any other relevant information related to the patient's health status and care plan. accurate documentation is crucial for patient care, as errors in clinical notes can lead to misdiagnosis, improper treatment, and potential harm to patients. by automating the process of error detection and correction, healthcare providers can ensure the integrity and reliability of patient records, ultimately improving the quality of care delivered. research indicates that a substantial proportion of adverse events in healthcare settings are due to errors in documentation, highlighting the need for effective error detection and correction mechanisms. in this task of medical error detection correction ben @xcite . we seek to address the problem * first author, equal contribution of identifying and correcting medical errors in clinical notes. this task had 3 subtasks. in subtask 1 (binary classification) researchers had to detect whether the clinical notes included a medical error or not. subtask 2 named span identification was to identify the text span associated with the error if a medical error exists. subtask 3 (natural language generation) was specifically to provide error-free text after making corrections if a medical error exists. in our approach, we initially conducted named entity recognition (ner) using gemini to identify words representing diseases or pathogens or suggestions in the text. after masking these identified words, we implemented the retrieval-augmented generation (rag) model on textbooks and external datasets. if the rag score fell below a certain threshold, we passed the input to our model, which was made by using 4-bit quantization on palmyra 20b and then fine-tuned the quantized palmyra model using the qlora technique on medqa data @xcite . if the word provided by palmyra or the rag model matched the word detected by ner, no error was detected. otherwise, if a different word was obtained, it was replaced with the masked word identified by ner. finally, the error sentence is mapped with the sentence id to get the output in the desired format. this approach helped us in getting a global rank 5 with an aggregate score of 0.73298. the aggregate score is calculated as the mean of rouge-1-f, bertscore, and bleurt . our model achieved r1f, bertscore, and bleurt scores of 0.70306, 0.74372 and 0.75217 respectively. the rest of the paper is organized as follows: review of related work and background information in sections 2 and 3 respectively, to provide context for our study. following this, we elucidate the system architecture in section 4 and describe the experimental setup in section 5. subsequently, we present our findings in section 6, discuss limitations encountered in section 7, and propose avenues for future research in section 8. finally, we have concluded our discussion in section 9. ## background the med dataset provided by organizers had 2 types of clinical notes -ms and uw. upon meticulous examination of the datasets , it became clear that the medical dataset which was divided into ms and uw clinical notes presented some unique difficulties. the ms sub- ms uw text id ms-val-108 uw-val-51 text a 3175-g (7-lb) female newborn is delivered at term. initial examination shows a flat perineal colonic atresia is confirmed when dark green discharge is coming out of the vulva. mr. <name/> has been noted to have documentation of thrombocytopenia @xmath0. plt 101 on admission. thrombocytopenia was present on admission (poa). sentences 0 a 3175-g 1 (7-lb) female newborn is delivered at term. 2 initial examination shows a . . . . 3 colonic atresia is confirmed . . . 0 mr. <name/> has been noted to have documentation of thrombocytopenia @xmath1. 1 plt 101 on admission. 2 thrombocytopenia was present on admission (poa). error flag 1 0 error sentence id 3 -1 error sentence colonic atresia is confirmed. . . na corrected sentence imperforate anus is confirmed when dark green discharge is coming out of the vulva. ## system description the subsequent sections provide a list of the submodules used. we will describe why and how each model was utilized, and assess its relevance to our problem statement. ## experimental setup we primarily used google colab notebooks for our workflow as well as for less computationally demanding tasks such as ner, eda, text masking, rag, etc. colab notebooks provide free access to a single t4 gpu (12gb ram, 8gb vram, 64gb disk space). however, running quantized llms or fine-tuning had much higher computational requirements, and we therefore used kaggle notebooks, which provide limited access to 2x t4 gpus (15 gb of vram each). please refer to for performing ner on text, we used the gemini api from google ai studio. it had a maximum query limit of 60 queries per second. since we were using gemini for ner as well as for rag, this became our bottleneck, which sometimes led the session to crash. to address this, we imposed a timeout after every few api calls as well as made frequent local saves to the inferred results. we implemented rag using the langchain framework, using gemini as our llm. for implementing retrieval in our knowledge base, we used gemini embeddings to populate our vector store, which was a locally created chromadb instance. ## result in our study, we employed a series of approaches aimed at enhancing the accuracy of our model. initially, we implemented the quantized palmyra approach, in which we tested the model that we built after the 4-bit quantization of palmyra-20b. this gave a modest aggregate score of 0.482371. however, recognizing the room for improvement, we continued to refine our methodology. building upon the quantized palmyra framework, we introduced the quantized+ qlora approach. in quantised palmyra, we fine-tuned using qlora on medqa data, which demonstrated a notable improvement, yielding an aggregate score of 0.55786. encouraged by this progress, we further augmented our model with the pure rag technique, resulting in a substantial enhancement in aggregate score to 0.63884. finally, through the integration of all three approaches-quantized, qlora, and rag-into our model, we achieved the highest aggregate score of 0.73298. the detailed scores for each approach are described in ## limitations the model struggles to give the correct output if the error is not related to a disease or pathogen. ner plays a crucial role in detecting pathogens or diseases from the text and therefore proves to be a bottleneck for accuracy since if ner fails to accurately determine the disease, pathogen, or suggestion, the result will not be accurate regardless of the robustness of the model. the rag approach fails for symptoms that are phrased very differently from those in the principal texts. finetuning on a larger dataset, which will contain richer and more diverse medical information, can improve the model performance. integrating multimodal information, such as images or structured data from electronic health records, alongside text data could provide richer context and improve error detection and correction accuracy. enhancing model robustness: the model can be made more robust against failures by having an end-toend architecture where individual modules like ner, error detection, etc. are not carried out independently. ## conclusion to conclude with this work for the mediqa-corr task at naacl, in clinicalnlp 2024, we investigated four approaches for detecting and correcting errors in clinical notes. our experiments demonstrated that the combined approach of quantised palmyra with rag achieved the best performance, with an aggregate score of 0.73298. however, a key limitation identified is the reliance on named entity recognition (ner). errors in ner can impact the overall performance of the system. looking towards the future, research efforts should focus on mitigating the dependence on ner. additionally, exploring alternative techniques and leveraging a larger, more comprehensive dataset holds promise for further improving the accuracy of error detection and correction in clinical notes. this will ultimately lead to a more robust and reliable system for enhancing the quality of clinical documentation.