M edi F act at MEDIQA - CORR 2024: Why AI Needs a Human Touch (2024)

Abstract: Accurate representation of medical information is crucial for patient safety, yet artificial intelligence (AI) systems, such as Large Language Models (LLMs), encounter challenges in error-free clinical text interpretation. This paper presents a novel approach submitted to the MEDIQA-CORR 2024 shared task k (Ben Abacha et al., 2024a), focusing on the automatic correction of single-word errors in clinical notes. Unlike LLMs that rely on extensive generic data, our method emphasizes extracting contextually relevant information from available clinical text data. Leveraging an ensemble of extractive and abstractive question-answering approaches, we construct a supervised learning framework with domain-specific feature engineering. Our methodology incorporates domain expertise to enhance error correction accuracy. By integrating domain expertise and prioritizing meaningful information extraction, our approach underscores the significance of a human-centric strategy in adapting AI for healthcare.

## introduction accurately identifying pathogens from textual descriptions of symptoms is crucial in effective healthcare management @xcite . however, existing datasets often present significant challenges that hinder reliable inferences and accurate pathogen identification, especially for rare diseases with limited data availability @xcite . one major challenge lies in the inherent linguistic ambiguities present within these descriptions. synonyms, homonyms, and polysemy (words with multiple meanings) can lead to confusion and misinterpretations @xcite . for example, the term "fever" could indicate a wide range of illnesses, making it difficult to pinpoint the specific pathogen without additional context. additionally, the distribution of diagnostic and pathogen information within the data can be imbalanced, with some diseases being vastly overrepresented compared to others. this imbalance can skew the model's performance and hinder its ability to accurately identify pathogens for less frequently encountered diseases @xcite . furthermore, incorporating sensitive diagnostic data for training llms raises significant ethical concerns regarding patient privacy and authorization requirements @xcite . moreover, pretrained llms often learn from vast amounts of generic text data, which might not be tailored to the specific domain of pathogenic research @xcite . this lack of domain-specific knowledge can hinder their ability to capture the nuances of rare disease entities and the intricate relationships between textual descriptions and underlying pathogens @xcite . existing approaches to medical text correction have explored various techniques, including rulebased systems like metamap (which utilizes predefined rules to map terms to standardized medical concepts) and machine learning algorithms like rnn-based models (trained to identify and correct errors based on patterns learned from training data) @xcite @xcite . however, these methods often struggle with the complexity of medical terminology, the inherent ambiguities of natural language, and the limitations of rule-based systems in capturing the ever-evolving nuances of medical language @xcite . while recent advancements in llms have shown promise in various natural language processing tasks like text correction, their application in medical diagnostics necessitates careful considera-tion due to the sensitivity of the data and the need for domain-specific knowledge. existing llmbased medical text correction approaches primarily address basic issues like typos and grammatical errors @xcite . however, they often fall short in addressing patient hallucinations, which can introduce factual errors and lead to misdiagnosis @xcite . additionally, fine-tuning these models on relevant datasets often yields limited improvements, with models producing generic corrections instead of medically accurate ones @xcite . this paper aims to present a methodology for automatically correcting single-word errors in clinical notes, submitted to the mediqa-corr 2024 shared task @xcite . the approach utilizes supervised learning with tailored feature engineering for the medical domain, emphasizing meaningful information extraction from clinical text data. two distinct strategies are employed: an extractive question-answering (qa) approach for observed error-correction pairs and an abstractive qa approach for unobserved relations. this framework addresses the following important research questions: 1) how can domain expertise be further integrated into the model to improve its accuracy and ability to explain its reasoning? 2) how can this approach be effectively utilized to assist human reviewers in the process of medical record correction, potentially improving efficiency and accuracy? 3) what ethical considerations are involved in using ai for automatic error correction in healthcare settings, such as potential bias, transparency, and accountability? ## methodology this paper introduces medifact-corr qa, a dataefficient approach for one-word error correction in clinical text paragraphs. medifact-corr qa leverages a two-stage process combining weakly supervised learning with pre-trained models to address labeled medical text data limitations. ## experimental setup and results this section details the experimental setup and evaluates the performance of our two-stage model for one-word error correction in clinical text paragraphs. ## discussion large language models (llms) have shown remarkable success in various natural language processing tasks, but their application in medical text correction faces unique challenges @xcite . our approach tackles the challenging task of correcting one-word errors in clinical text paragraphs. unlike llms that rely solely on statistical patterns learned from vast amounts of text data, our approach utilizes features specifically tailored to the medical context. this allows the model to leverage domain knowledge and prioritize terms. the example demonstrating the limitations of llms and the strengths of svms with tf-idf can be added as a separate paragraph in the same section, following the current paragraph. example paragraph: 'a 5-year-old male presents with complaints of a painful mouth/gums, and vesicular lesions on the lips and buccal mucosa for the past 4 days. he is unable to eat or drink due to the pain and reports muscle aches. vital signs: t 39.1°c, hr 110, bp 90/62 mmhg, rr 18, spo2 99%. physical examination reveals vesicular lesions on the tongue, gingiva, and lips, with some ruptured and ulcerated, and palpable cervical and submandibular lymphadenopathy. patient is diag-3 medifact-svm models are available: https://github. com/nadiasaeed/medifact-mediqa-corr-2024 nosed with an [mask] infection.' while a fine-tuned distillbert model predicted a general term like 'goat' or 'highlander' @xcite . on the other side, our svm model trained with tf-idf utilizes domain knowledge through feature weights @xcite . features like 'vesicular lesions', 'lips', and 'gingiva' receive high weights, guiding the model towards the medically accurate prediction of 'hsv-1' due to its alignment with the clinical context." our journey focused on error detection and correction within clinical text data. while transformer-based models are powerful, their limitations in interpretability, data requirements, and over-fitting prompted us to explore an alternative: svms with tf-idf features. unlike many models, svms offer valuable insights through feature weights @xcite . features were designed to recognize specific medical terms, abbreviations, and entities like drug names, diagnoses, and anatomical locations. rules and patterns observed in common errors were translated into features @xcite . features captured aspects like sentence structure, negation markers, and temporal inconsistencies, which can indicate factual errors like incorrect dates or inconsistent medication names. the provided dataset posed a unique challenge due to pre-defined sentence indices that deviated from standard newline ("\n") splitting @xcite . to address this challenge, we compared detected errors' content with the dataset's available sentences. the index reported in the "error sentence index" column was predicted as the starting digit of the most similar sentence. therefore, we must recognize that inherent dataset issues influenced our final score. these challenges underscore the significance of high-quality data for training machine learning models. in our submission, we investigated three key outcomes in an alternative setting. in the first and second scenarios, utilizing a qa model instead of the static correction model of svm resulted in an improved r1f score from 0.342 to 0.454. this enhancement underscores the effectiveness of employing a qa model for error correction tasks. moreover, the accuracy of error sentence detection significantly increased from 0.39 to 0.6 by utilizing the starting digit of the most similar sentence in the pre-defined index of sentences within given samples. this improvement stemmed from addressing an index problem, specifically by selecting the index from the upper part of the sentence. ## future work our initial success with svms for pathogen identification in clinical text data paves the way for further exploration using llms. however, llms pose unique challenges. data scarcity, particularly in the specific medical domain, could be a significant hurdle @xcite . limited data restricts the use of a separate validation set. future work will explore acquiring more data and data augmentation to enhance model generalizability. techniques like data augmentation and transfer learning from pre-trained medical llms might be crucial to overcome this limitation. ethical considerations are paramount, and mitigating biases within the training data is essential. furthermore, ensuring interpretability through techniques like attention mechanisms is vital for trust and acceptance in healthcare settings. finally, for practical implementation, we need to explore computationally efficient llm architectures or develop task-specific models focused on pathogen identification. continuous evaluation through techniques like active learning and performance monitoring will be crucial for maintaining a robust, ethical, and interpretable system in realworld clinical text analysis.