V erba N ex AI at MEDIQA - CORR : Efficacy of GRU with B io W ord V ec and C linical BERT in Error Correction in Clinical Notes (2024)

Abstract: The automatic identification of medical errors in clinical notes is crucial for improving the quality of healthcare services.LLMs emerge as a powerful artificial intelligence tool for automating this task. However, LLMs present vulnerabilities, high costs, and sometimes a lack of transparency. This article addresses the detection of medical errors through the fine-tuning approach, conducting a comprehensive comparison between various models and exploring in depth the components of the machine learning pipeline. The results obtained with the fine-tuned ClinicalBert and Gated recurrent units (Gru) models show an accuracy of 0.56 and 0.55, respectively. This approach not only mitigates the problems associated with the use of LLMs but also demonstrates how exhaustive iteration in critical phases of the pipeline, especially in feature selection, can facilitate the automation of clinical record analysis.

## introduction large language models (llms) demonstrate promise in tackling unseen tasks with notable competencies. however, these models exhibit a fundamental vulnerability. llms are costly to train and utilize: their cost has increased 10 to 100-fold since 2015 and must be run on giant compute clusters. the training data used for corporate models is a closely guarded secret that lacks transparency @xcite . additionally, the success of llms has led to certain online content being generated entirely by these models, which are susceptible to producing non-factual information. in specialized domains, online information can be unreliable, detrimental, and contain logical inconsistencies that impede the models' reasoning ability. nevertheless, most prior research on common sense detection has concentrated on the general domain. @xcite . in this context, our study focuses on the challenge of identifying common sense errors in clinical notes. unlike correcting these errors, which requires a deep understanding and specific knowledge of the medical field, identification is a crucial first step that demands the models' ability to recognize inaccuracies and anomalies in the text. this work explores how advanced natural language processing (nlp) technologies, such as gru with bioword-vec, and especially clinicalbert @xcite , can be useful for analyzing unstructured medical texts. our methodological approach involves a comprehensive comparative analysis among these models, highlighting their efficiency in identifying errors in clinical notes, underscoring the relevance of adapting model training to the peculiarities of medical data. we seek to demonstrate that, through specialization and fine-tuning of these llms models, it is possible to significantly improve their ability to detect erroneous or missing information, crucial for diagnosis and treatment in the clinical setting. this study not only aims to demonstrate the capabilities and limitations of advanced models in specialized medical contexts but also to emphasize the importance of integrating specialized knowledge within llms to optimize the reliability and usefulness of clinical notes in medical practice. this document is described as part of our participation in the shared task medical error detection and correction of the association for computational linguistics @xcite . ## related work in recent advances in the field of nlp, the ability to identify common sense errors in clinical notes poses a significant challenge and represents an opportunity to improve the quality of healthcare. the relevance of this study lies in exploring the applicability of advanced nlp models for the accurate detection of inaccuracies in medical records. these models constitute a promising advance over the inherent limitations off llms especially those arising from the quality and diversity of their training datasets @xcite . llms often require domain-specific adaptations to effectively handle specialized tasks due to these limitations @xcite . moreover, models like clinicalbert have been shown to significantly improve their performance in interpreting clinical language by adapting to specific contexts @xcite . nlp has the capacity to derive meaningful insights from unstructured data, specifically in the domain of categorizing incident reports and adverse events. understanding the nature and reasons behind these incidents is crucial for analyzing adverse events. if nlp can enable the extraction of these insights from larger datasets, it has the potential to enhance learning from adverse events in the healthcare field. @xcite . given the complexity of clinical notes and the necessity for a high degree of precision in their analysis, this study is grounded in the review of previous research that has addressed similar issues in the domain of medical text classification. a relevant study focused on clinical text classification using rule-based features and knowledge-guided convolutional neural networks, leveraging trigger phrases and unique medical concepts (cuis) from the unified medical language system (umls) to enhance classification accuracy in class-imbalanced situations @xcite . this approach underscores the effectiveness of integrating deep learning with explicit medical knowledge, emphasizing the importance of adapting model training to the specificities of clinical data. additionally, a comparative investigation evaluated various deep learning models, including convolutional neural networks (cnn), recurrent neural networks (rnn), gru, long short-term memory (lstm), bidirectional long short-term memory (bi-lstm), and a transformer encoder, in their ability to handle unstructured medical note texts affected by different levels of class imbalance @xcite . this analysis provides a critical perspective on the variability in model performance in the face of the unique challenges posed by medical data, highlighting the need for more specialized and adaptive approaches. these studies and similar efforts outline the current state of using advanced nlp technologies in medical text classification. the present study draws inspiration from these research endeavors to advance understanding of the application of specific nlp models in error identification in clinical notes. in doing so, we aim to contribute to the field by providing valuable insights for future research and practices in this essential domain. ## system description in the system description of our study, we address the implementation of an advanced predictive model specifically designed for detecting errors in clinical notes. this model relies on two fundamental pillars of nlp: gru and the clinicalbert architecture @xcite . the formulation of our central hypothesis questions the effectiveness of lexical and contextual features obtained through these nlp technologies to identify inaccuracies within clinical texts. we propose two main methodological strategies. the first strategy implements gru to extract lexical features, leveraging its ability to process complex temporal dependencies in the data @xcite . this aspect is reinforced by the use of biowordvec, which provides detailed vector representations of medical terms, thereby facilitating the capture of the semantic complexities of clinical language. the adaptability of gru models to variable-length sequences proves particularly useful for analyzing medical texts, where critical information may be irregularly distributed throughout the document @xcite . the second strategy focuses on harnessing clin-icalbert, a model known for its ability to weigh the relevance of words through attention mechanisms, thereby enabling a deep understanding of the context in which medical terms are embedded. this approach significantly benefits from transfer learning, adapting previously acquired knowledge from extensive medical text corpora to fine-tune the model for our specific task. the synergy between gru and clinicalbert enables a comprehensive analysis of the texts, evaluating not only coherence but also the semantic accuracy of the clinical content @xcite . clinicalbert exhibits superior performance in identifying significant connections between medical concepts, a validation corroborated by medical experts @xcite .. this model has surpassed several benchmarks in predicting 30-day hospital readmissions, using discharge summaries and notes from early intensive care units, covering multiple clinically relevant metrics @xcite . the attention weights generated by clinicalbert facilitate the interpretation of predictions, providing a deeper understanding of the context in which medical terms are embedded. we have released the model parameters and training scripts to encourage further research in this field. thanks to its flexible structure, clini-calbert can be easily adapted to other predictive tasks with minimal engineering effort, making it ideal for studies requiring detailed analysis of clinical language @xcite . based on the outlined strategies, we configure a detailed training system as depicted in figures 1 and 2 of the study. this system unfolds through a sequence of well-defined stages: data ingestion and preliminary cleaning, generation of training instances, extraction of both lexical and contextual features, followed by the classification phase, and finally, model evaluation. this process ensures comprehensive treatment of clinical notes, optimizing error detection through the joint evaluation of long temporal dependencies and detailed contextual analysis. this approach highlights not only the relevance of incorporating advanced nlp tools in the assessment of clinical texts but also the potential of these technologies to progress towards a higher degree of accuracy and reliability in medical documentation. ## data description the dataset provided by mediqa-corr @ naacl-clinicalnlp 2024 @xcite offers a comprehensive collection of medical texts, each corresponding to a clinical case report. this dataset stands out for its structured and detailed content, tailored for facilitating the analysis and identification of medical errors. below are the key features of this dataset: this dataset represents a valuable tool for research in the field of nlp applied to medicine, especially in tasks related to the identification and correction of errors in clinical texts. the richness and specificity of the data facilitate the development and evaluation of advanced nlp models, as addressed in this study, providing a solid foundation for detailed analysis and improvement of the quality of clinical notes. ## embeddings in the process of generating embeddings for our analysis, we applied meticulous preprocessing to the provided data. this preprocessing consisted of a series of essential steps to ensure the quality and uniformity of the text, including correcting encoding errors and normalizing medical terms and units of measurement. this preliminary treatment of the texts is crucial to mitigate variations and ensure the integrity of the analyzed data. subsequently, we focused on transforming these normalized texts into vector representations using the biowordvec model. this model, specifically trained on extensive medical corpora, was selected for its ability to accurately capture the semantics and clinical context of the terms used in the notes. by converting the texts into 200-dimensional vectors, representations of unrecognized words were adjusted using @xmath0, following a standardized approach for sequence length. this textto-embeddings transformation procedure is essential for subsequent analysis using nlp techniques. we used biowordvec based a previous study, which findings across five models utilizing various word embeddings indicate that biowordvec embeddings marginally enhanced the bi-lstm model's performance for certain datasets. overall, models incorporating biowordvec embeddings exhibited slightly superior performance compared to those utilizing glove embeddings @xcite . through tokenization and sequence adjustment, we prepared the data for processing by advanced models such as gru and clinicalbert. these models require structured and coherent inputs to effectively interpret the information contained in the clinical notes and thus identify possible errors. the meticulousness of this approach highlights the importance of preprocessing in nlp-supported clinical research. by transforming clinical notes into contextualized embeddings, we facilitate deep and accurate analysis by nlp models, enhancing error detection. this process not only enhances the analytical capability of the models but also underscores the value of rigorous data preparation in the field of artificial intelligence applied to medicine. ## data transformation after normalizing the data, we proceeded with its segmentation into training and test sets, adjusting this split according to the specific model to be used and experimenting with different partitions to always seek optimal accuracy. for the analysis with gru, we selected an 80-20 split for training and testing, respectively, while for the evaluation using biowordvec and clinicalbert, the distribution was adjusted to 70-30. this differentiation allowed us to adapt the learning and validation process to the peculiarities of each model, optimizing their ability to analyze and understand complex clinical texts. this meticulous preparation and segmentation figure 2: model bio clinical ## feature extraction the process of extracting lexicographic features involved analyzing fundamental aspects of the text, such as the use of specific terms and the overall semantic structure of the clinical notes. this included evaluating polarity and the frequent use of parts of speech, which are indicative of the tone and intention of the medical text. through this analysis, we sought to better understand how lexicographic features can influence the presence of errors within the notes. for the gru-based model, we adjusted the class weights to address the imbalance in our data, using the number of unique classes derived from the training set. this adjustment was crucial for training a balanced model capable of effectively classifying texts based on the presence or absence of medical errors. the gru model was configured with layers specifically designed to capture and analyze complex temporal dependencies within clinical texts, including regularization layers to prevent overfitting and optimize overall performance. simultaneously, for the implementation based on clinicalbert, we proceeded with data tokenization and preparation using the autotokenizer from 'emilyalsentzer/bioclinicalbert'. this preparation was essential to adapt our clinical notes to the format required by clinicalbert, allowing the model to process and classify the texts efficiently. the training of the model focused on binary classification of texts, training on contextualized representations generated to identify the presence of errors with high precision. the training of the gru and clinicalbert models was conducted under carefully selected configurations to optimize their learning and evaluation on the dataset. these configurations included defining the number of epochs, batch size, and learning rate, which are fundamental elements for the success of the training. ## settings in the setup of the study, specific adjustments were made to the hardware and software parameters to optimize the analysis of the gru and clinical-bert models. these adjustments included the optimization of processors and the allocation of execution threads, essential for the efficient processing of the clinical dataset. additionally, differentiated configurations were implemented in the software environment to adapt to the peculiarities of each model. this involved optimizing data loading, preprocessing, and embedding generation, ensuring that both gru and clinicalbert operated under optimal conditions for text analysis. adapting the computational environment allowed for maximizing the capabilities of each model, facilitating a thorough and precise analysis of clinical texts. the computational infrastructure was also configured to log and store the highest performing features and classifiers during the experimental phase. this systematic approach allowed for continuous monitoring of model performance, providing a solid foundation for iteration and enhancement of analysis strategies. this detailed setup reflects the methodical and rigorous approach adopted for the preparation and execution of the nlp models. by optimizing computational resources and adapting the software, the necessary conditions were established for an effective evaluation of the models' ability to identify errors in medical documentation. ## experiments and analysis of results comprehensive evaluations of multiple natural language processing models were conducted using the dataset provided by mediqa-corr @ naacl-clinicalnlp 2024, with the goal of identifying those with the best performance in detecting errors in clinical notes . these experiments not only allowed for the adjustment of model configurations but also served to identify optimal techniques that significantly contribute to the analysis of medical texts. among the evaluated models, gru and bioclinicalbert proved to be the most effective across various metrics and scenarios, which is why they were selected for further detailed analysis. during the initial evaluations, models such as rf, roberta, bert, and cnn were also tested. hyperparameters for these models were adjusted to obtain better results, revealing their potential when dealing with larger datasets @xcite . the implementation of rf and cnn models highlighted the importance of feature identification and automatic feature extraction, respectively @xcite . moreover, the use of bert models leveraged the transformer architecture to pre-train language representations, enhancing the understanding of context and semantics in clinical terms @xcite . this extensive evaluation facilitated the refinement of strategies and parameters for each model, aiming to maximize their accuracy in classifying texts based on the presence of errors. throughout multiple iterations in the preevaluation phase, strategies and parameters for each of these selected models were refined with the goal of maximizing their ability to classify texts accurately based on the presence of errors. standard competition metrics, with a special emphasis on accuracy (acc), were employed to measure the performance and effectiveness of the developed systems. the experiments revealed notable differences in the efficacy of the gru and bioclinicalbert models for analyzing the medical corpus. while gru excelled in its ability to process text sequences and capture temporal dependencies, bioclinicalbert proved to be particularly effective in understanding the context and specific semantics of clinical terms. this distinction underscores the complementarity of the models in handling complex medical texts. the results, summarized in ## result test among the evaluated models, clinicalbert and gru stood out for their robust performance across various metrics and were selected for further detailed analysis. after rigorous validation, which included reviewing performance and learning curves, the selection of clinicalbert and gru was based on a rigorous analysis of their capacity to process and analyze complex clinical texts, showing notable superiority in identifying errors in medical documentation. the validation of these models confirms the effectiveness of our selection strategy and highlights the importance of exploring in depth how these models can contribute to improving the analysis of clinical notes in the future. ## discussion and conclusion the meticulous selection of embeddings and nlp models, specifically gru and clinicalbert, is crucial for the accurate analysis of clinical texts, as evidenced in our findings. these decisions are vital for optimizing error detection in clinical notes. however, there is a need to expand experimentation with a broader spectrum of models and embeddings to validate their effectiveness in specific clinical contexts. the analysis of the results, presented in despite considering the use of advanced llms like gemini or chatgpt-4, this study highlights the efficacy of alternative models such as clini-calbert and gru. this preference is due not only to their competent performance but also to their specific adaptability to the demands of clinical texts. this approach is crucial in environments this study demonstrates that advancements in nlp can significantly contribute to the clinical field, although it also highlights the ongoing challenge of adapting these technologies to the specificities of medical language and data. nlp models, even in the face of accuracy challenges, prove to be valuable tools when carefully adjusted based on a deep understanding of the context and specific objectives of the task. for future research, feature selection is highlighted as the primary strategy. it is suggested to focus on the development and application of advanced methodologies for feature extraction and selection, with the aim of refining the analytical capabilities of models for the precise processing and understanding of medical texts. this methodological approach not only anticipates an increase in the accuracy of models for anomaly detection and error identification in clinical documentation but also promises to deepen our understanding of the adaptation and optimization of nlp techniques for specific needs within the healthcare domain. in conclusion, this study significantly contributes to the field of nlp applied to the medical domain, promoting the continuous innovation and optimization of models that, through meticulous choice and configuration of features, have vast potential to elevate the quality of clinical documentation. a notable finding is the moderate impact that pretrained embeddings have on model performance, indicating that the integration and thorough exploration of these pre-trained tools can be crucial for amplifying the effectiveness of nlp in clinical contexts. this constant adaptation and improvement of technologies promise to advance towards optimizing the practical utility of nlp models, thereby contributing to improving the standards of care and documentation in the healthcare sector.