Enriching Electronic Health Record with Semantic Features U tilising P retrained Transformers (2023)

Abstract: Electronic Health Records (EHRs) have revolutionised healthcare by enhancing patient care and facilitating provider communication. Nevertheless, the efficient extraction of valuable information from EHRs poses challenges, primarily due to the overwhelming volume of unstructured data, the wide variability in data formats, and the lack of standardised labels. Leveraging deep learning and concept embeddings, we address the gap in context-aware systems for EHRs. The proposed solution was evaluated on the MIMIC III dataset and demonstrated superior performance compared to other methodologies. We addressed the positive impact of the latent feature combined with the note representation in four different settings. Model performance was evaluated using a case study conducted with BertScore, assessing precision, recall, and F1 scores. The model excels in Medical Natural Language Inference (MedNLI) with an 89.3% accuracy, further boosted to 90.5% through retraining the embeddings using International Classification of Diseases (ICD) codes, which we formally designate as ClinicNarrIR. The ClinicNarrIR was tested with 1000 randomly sampled notes, achieving an N DCG@10 score of approximately 0.54 with accuracy@10 of 0.85. The study also demonstrates a high correlation between the results produced by the proposed representation and medical coders. Notably, in all evaluation cases, the optimal base pretrained model that emerged was BlueBERT.

## introduction in healthcare, electronic health records (ehrs) have gained widespread adoption due to their potential to enhance patient care, minimise medical errors, and promote efficient communication among healthcare providers. however, extracting relevant information from these extensive records remains a challenging and time-consuming task for clinicians. the primary challenge, besides the lack of labeled data, is also the absence of an effective and context-aware system that can semantically retrieve valuable information from ehrs based on practitioner queries @xcite . current approaches often rely on structured data or keyword matching/expansion and overlook contextual nuances, clinical history, and natural language intricacies. retrieving the information is essential beyond patient care; it is vital for secondary clinical use cases, including clinical trial matching @xcite @xcite , drug repurposing @xcite , quality assurance @xcite , clinical decision support @xcite , research access @xcite , and population health management @xcite . these applications demand a comprehensive approach that goes beyond traditional methods, contributing not only to improved patient care but also to significant advancements and breakthroughs in the field. to address this gap, our research aims to take a significant step towards the development of an automated clinical information retrieval system. the primary objective is to design a context-enhanced network capable of semantically searching electronic health records (ehrs), returning relevant clinical notes in response to practitioner queries, and embedding clinical narratives with semantic features. a clinical narrative refers to the free-text entries made by healthcare practitioners to track a patient's progress from admission to discharge @xcite . on the other hand, a clinical concept refers to specific entities within the clinical notes, which could include diagnoses or diseases, serving as semantic features. the research explores deep learning pretrained models to represent these embeddings and investigates the impact of an additional element (medical concept) through four different settings. additionally, it enhances medical language representation by training a network to learn the clinical narrative vector as the ground truth query. these contributions aim to advance the field by improving clinical information extraction and testing model performance. this comprehensive approach ultimately leads to enhanced clinical decision-making and patient care the utilisation of medical natural language processing (mlp) in processing ehrs is essential due to the unique characteristics of clinical notes @xcite . these notes often contain abbreviations and deviate from standard english grammar rules @xcite . in contrast, discharge summaries adhere to more formal language standards, making techniques developed for non-medical text sources potentially applicable @xcite . information extraction (ie) in nlp automates the identification of concepts, entities, events, and their relations in free text @xcite . ie encompasses various subtasks, including named entity recognition (ner), co-reference resolution, and relation extraction @xcite . ner is especially significant in the medical field, where it is used to classify entities such as genes, medicines, and diseases within unstructured text @xcite . conditional random fields (crf), a wellknown model, contributes to concept extraction by considering words before and after the entity @xcite bert, a leading nlp model @xcite , has gained traction in the health informatics field with the development of domain-specific versions like clinicalbert @xcite . clinicalbert, pretrained on clinical text from the mimic-iii database @xcite , provides word embeddings for research without introducing novel training techniques. it outperforms general bert in clinical tasks like named entity recognition (ner, i2b2) and medical natural language inference @xcite . additionally, biobert is another domainspecific bert model, trained on biomedical research data sourced from @xcite . its specialization in biomedical texts enhances performance across various biomedical nlp tasks @xcite . key research themes in ehr include information extraction, ehr representation, outcome prediction, computational phenotyping, and clinical data de-identification @xcite @xcite @xcite . information extraction involves concept extraction, temporal event identification, and abbreviation expansion, primarily using clinical notes. ehr representation uses medical codes to represent concepts/patients. outcome prediction includes applications like hospital readmission and mortality prediction @xcite . semehr, a tool for semantic search in a clinical context, uses ontology to locate mentions of biomedical topics in ehrs @xcite . unsupervised embedding is advantageous, given the challenges of obtaining human labels for clinical records. combining medical concepts with clinical narrative, as demonstrated by @xcite , outperforms other unsupervised approaches in tasks like mortality prediction and patient relatedness @xcite . in the upcoming sections, the methodology will be detailed, discussing indexing, retrieval, and the value of retraining embeddings. subsequently, the evaluation includes a case study, mednli assessment, and correlation analysis, followed by discussions of pretrained models, and limitations. ## methodology in siamese bert, a fixed-sized sentence embedding is created by adding a pooling layer to the bert output, facilitating the comparison of input similarity. siamese neural networks consist of multiple identical subnetworks that share parameters and configurations, aiding in various applications @xcite . the document indexing phase, as illustrated in figure 1 , leverages deep learning and keywordbased methods, such as tf-idf (term frequency -inverse document frequency), to enable the semantic and syntactic searchability of clinical notes at scale. concept embeddings provide detailed information about clinical entities within the notes, while narrative embeddings capture their overall relevance and context. by integrating these embeddings, the retrieval stage can identify notes with similar clinical concepts, thereby enhancing the relevance of results for clinicians. during indexing, clinical text n is transformed into vectors [n 1 , n 2 , .., n n ], where each note is embedded as [n e 1 , n e 2 , .., n e n ] and combined with concept embeddings [ce 1 , ce 2 , ...., ce n ]. the concepts were extracted using the medcat tool @xcite and filtered, retaining symptoms, past diseases, and disorders. this process ensures that the relevance ranking model prioritises clinically significant information while reducing noise and enhancing semantic understanding. by considering the clinical context and incorporating these filtered concepts, the system improves information retrieval, aiding healthcare practitioners in making informed decisions and delivering high-quality patient care. to evaluate the effectiveness of the network, particularly the added value of concept embeddings, performance was assessed by translating notes into embeddings and exploring the impact of adding concepts. different settings were examined: using only clinical narrative (setting 1), using multiple concept embeddings (setting 2), combining concept embeddings and averaging them into a single vector, represented as [ce 1 . . . ce z ]/z where z is the number of concepts (setting 3), and combining the note with concepts (setting 4). in the retrieval stage depicted in figure 2 a sorted set of relevant notes is returned from the corpus by computing the cosine similarity between query q and note n. during retrieval using setting 2, equation 2 is computed and produce multiple vectors with a cosine similarity score that measures the likelihood of a concept being relevant to query q. a higher score indicates greater similarity to the keyword query. where n is the number of notes and z number of concepts in each note. in setting 3, the combined concepts [ce 1 . . . ce z ] for all n notes, denoted as an 1 ..an n . the cosine similarity is calculated between query and ∀@xmath0, and the n with the maximum similarity score will be retrieved. the approach considers various strategies, such as multiplication, or weighted sum, and averaging to combine tf-idf vectors with embeddings. in the following section, we will conduct a case study on a female patient records, evaluate mednli performance, and analyse physician and coder correlation. these assessments will offer insights into the effectiveness of different settings in real-world clinical applications. ## experiments and evaluation evaluating unsupervised problems, especially in the context of information retrieval (ir), presents distinct challenges. the evaluation involves a case study where we apply bertscore to assess precision, recall, and f1. we also measure accuracy in mednli using the proposed approach and its variation. additionally, we investigate the correlation with physicians/coders in drawing inspiration from @xcite evaluation. ## challenges & limitation in healthcare-based deep learning and information retrieval, several challenges emerge. pretrained models demand high gpu memory, and there's a delicate balance to be struck between sequence length and batch size, essential for efficient training. training embeddings for clinical notes that are associated with numerous icd codes can result in suboptimal similarity between the note and the icds. this is because the training process may introduce conflicting rewards in one batch and penalties in another batch, potentially affecting the expected level of similarity. inaccurate icd code assignments by coders to a note can also contribute to this issue. designing hard negatives sample for icds may offer a solution. the choice of evaluation metrics is critical, as they must align with specific research goals and be mindful of their limitations. substantial gpu memory, preferably over 32 gb, is required for effective work, which can pose challenges for those with limited access to highend hardware. training data may exhibit variations in query exposure, impacting result relevance, necessitating strategies for consistency. beyond technical concerns, researchers struggle with limitations on data availability imposed by patient privacy regulations, such as the health insurance portability and accountability act (hipaa) and the general data protection regulation (gdpr). additionally, they encounter challenges in overcoming issues of data scarcity, quality, and labour-intensive annotation processes. navigating these complexities, which span technical, ethical, and data-related dimensions, is essential to advancing their work in this crucial field. ## conclusion this research introduces a novel approach to enhance clinical information retrieval from electronic health records (ehrs) by employing a contextenhanced network that leverages deep learning and concept embeddings. it demonstrates the effectiveness of various settings and the importance of integrating clinical narratives with concepts. notably, the study highlights the model's practical application through a case study of a patient with pneumonia infection. the findings reveal that setting 4, incorporating concept embeddings, consistently outperformed other configurations. moreover, retraining the embeddings using icd significantly boosted accuracy to 90.5% in the mednli dataset. the clinicnarrir model achieved an n dcg@10 score of 0.54, showing some potential in retrieving clinically relevant information from ehrs, ensuring that the most critical patient data is readily accessible. the analysis of physician and coder correlation demonstrated a strong correlation coefficient of 0.72 with coder ratings (p-@xmath1.91e-06). furthermore, the effectiveness of different pre-trained transformers was investigated, with bluebert consistently delivering superior results in all cases. leveraging mimic-iii in this study proves advantageous to the research community, providing realistic data, benchmark status, open access, and ethical de-identification. this contribution establishes a valuable benchmark for fellow researchers. however, this research acknowledges several challenges and limitations in healthcare-based deep learning and information retrieval, including the need for substantial gpu memory, data privacy regulations, and data scarcity. despite these challenges, this research paves the way for more accurate and efficient clinical decision-making, benefiting both healthcare practitioners and patients. the comprehensive assessments and results presented herein offer valuable insights into the effectiveness of different settings in real-world clinical applications, reinforcing the significance of this work.