HSE NLP Team at MEDIQA - CORR 2024 Task: In-Prompt Ensemble with Entities and Knowledge Graph for Medical Error Correction (2024)

Abstract: This paper presents our LLM-based system designed for the MEDIQA-CORR @ NAACL-ClinicalNLP 2024 Shared Task 3, focusing on medical error detection and correction in medical records. Our approach consists of three key components: entity extraction, prompt engineering, and ensemble. First, we automatically extract biomedical entities such as therapies, diagnoses, and biological species. Next, we explore few-shot learning techniques and incorporate graph information from the MeSH database for the identified entities. Finally, we investigate two methods for ensembling: (i) combining the predictions of three previous LLMs using an AND strategy within a prompt and (ii) integrating the previous predictions into the prompt as separate ‘expert’ solutions, accompanied by trust scores representing their performance. The latter system ranked second with a BERTScore score of 0.8059 and third with an aggregated score of 0.7806 out of the 15 teams’ solutions in the shared task.

## introduction medical records play a crucial role in healthcare systems as they capture essential patient information, including diagnoses, treatments, and outcomes. medical texts are characterized by complex terminology, context-specific knowledge, and significant implications. detecting and rectifying errors within clinical notes necessitates domain expertise and reasoning. this task presents a complex challenge that demands precise analysis and understanding of the medical domain. in recent years, large language models (llms) have revolutionized the field of natural language processing (nlp) by demonstrating unprecedented performance across a wide range of tasks. these models, often based on transformer @xcite , have become the cornerstone of modern nlp research @xcite . llms excel in key areas such as semantic un-derstanding and contextualization @xcite , multimodal capabilities @xcite , few-shot and zero-shot learning @xcite , as well as various medical applications including disease diagnosis @xcite , drug discovery @xcite , and medical records processing @xcite . automated fact-checking has garnered significant attention due to the escalating challenge posed by misinformation. traditionally, fact-checking has relied on manual verification conducted by human experts, primarily focusing on generaldomain texts like wikipedia articles and news reports @xcite . recently, llms have offered the capability to analyze false statements and provide an assessment of their factual accuracy by leveraging their pre-trained knowledge and contextual understanding @xcite @xcite . several methodologies have been proposed to enhance the overall performance in llms, and the most notable ones are chain of thought (cot) @xcite . in this work, we utilize several key approaches for medical records correction using llms (see figure 1 ). these approaches include entity extraction and normalization @xcite @xcite , few-shot learning techniques @xcite , graph-based knowledge incorporation @xcite , and ensembling strategies @xcite . we investigate the application of these approaches to enhance the accuracy of medical error correction. the paper is organized as follows. section 1 presents shared task and data overview. we describe our approach with three key components and state-of-the-art (sota) models in section 2. experiments with baselines and our model are presented in section 3.3.4. finally, we discuss the results and conclude the work in sections 4 and 5, respectively. figure 1 : the system overview. the process can be described as follows: the system begins by receiving medical text as input. initially, a prompt template is utilized, supplemented with a small number of few-shot examples (either 2 or 5). the named entity recognition (ner) model is then employed to identify and extract named entities within the large language model's context. subsequently, potential replacements for these extracted entities are sought within the medical subject headings (mesh) thesaurus. the prompt, enriched with these replacements, is passed to the selected openai model. finally, the model's output is returned and stored in the prediction file. this constitutes the overall operation of the system. ## task and data overview the mediqa-corr 2024 shared task @xcite focuses on analyzing snippets of clinical text to address specific subtasks related to medical error detection and correction. these subtasks include: ## method the error correction method of figure 1 , proposed in the current work, is straightforward and consists of three major steps: we first prepare the data to make predictions: extract named entities from texts, and search for the term replacements. then we form the prompt for the model from the template, add a few examples and additional data (ner results, mesh terms), and then use llms to make predictions. ## experiments and results the evaluation results of our error correction systems are shown in the methodology delineated herein possesses the potential for expansion and further refinement through the incorporation of techniques such as the knowledge graph, promptkg @xcite , the meta-prompting approach, and the chain of thought (cot) approach. additionally, the integration of specialized models, specifically designed for error detection and error span identification, into the model pipeline could be achieved directly by utilizing the chaining techniques (e.g. langchain). this would serve to enhance the robustness and accuracy of the overall system. ## conclusion in this work, we have addressed the issue of identifying and resolving error text in biomedical texts. we have proposed a system for the mediqa-corr shared task by utilizing prompting, ensembling techniques, and llms. our approach demonstrates that the problem can be solved using ordinary gpt models without pre-training, relying solely on in-context learning, along with the ner model and additional mesh knowledge graph data. by employing an in-prompt ensemble of llms as experts and incorporating data from the mesh knowledge graph and ner results, we achieved a high task aggregated score of 0.78059, securing the 3rd position on the official competition leaderboard. our results highlight the effectiveness of our proposed prompting approach while also indicating areas for future improvement. utilizing more advanced tools like full-scale rag and fine-tuned biomedical llms could potentially enhance the quality of error correction. in addition, we plan to make all our code and data publicly accessible shortly after the publication of our paper.