Roux-lette at “Discharge Me!”: Reducing EHR Chart Burden with a Simple, Scalable, Clinician-Driven AI Approach (2024)

Abstract: Healthcare providers spend a significant amount of time reading and synthesizing electronic health records (EHRs), negatively impacting patient outcomes and causing provider burnout. Traditional supervised machine learning approaches using large language models (LLMs) to summarize clinical text have struggled due to hallucinations and lack of relevant training data. Here, we present a novel, simplified solution for the “Discharge Me!” shared task. Our approach mimics human clinical workflow, using pre-trained LLMs to answer specific questions and summarize the answers obtained from discharge summaries and other EHR sections. This method (i) avoids hallucinations through hybrid-RAG/zero-shot contextualized prompting; (ii) requires no extensive training or fine-tuning; and (iii) is adaptable to various clinical tasks.

## introduction clinicians spend considerable amount of time navigating vast amounts of electronic health records (ehrs) information, often spending 2-3 times more time interacting with ehrs than with patients @xcite . much of this time is spent on manual information retrieval and stylized summarization of relevant content, referred to as the charting burden. the "discharge me!" shared nlp task @xcite aims to alleviate this burden by automating the generation of "brief hospital course" (bhc) and discharge instructions from ehrs in the mimic iv dataset @xcite . these sections are critical but time-consuming, requiring synthesis, interpretation, and summarization of data from various parts of the patient's medical charts include admission notes, progress notes, lab results, and radiology reports. automated approaches such as rule-based systems and supervised machine learning models have been explored but often struggle with the complexity and variability of clinical data. these challenges lead to issues like hallucinationsplausible but incorrect information-and incomplete summaries due to missing data in the training sets. despite significant progress in machine learning for nlp tasks, achieving contextualized, relevant, and accurate summarization remains challenging. supervised ml approaches face difficulties in domain adaptation, and fine-tuning does not fully address these issues @xcite [3] @xcite . moreover, and while some progress was reported using fine tuning (e.g., @xcite ), the approach we describe here achieves better results without bearing the enormous cost of fine tuning. the lack of training data and llms' limitations in handling domain-specific data further complicate the problem. additionally, different users require different types of information extracted from ehrs, making contextual summarization essential. our solution uses a question-based approach to treat this summarization task as a context-aware and domain-specific question-answering process (see figure 1 ). the summaries generated using our pipeline answer specific questions posed by subject-matter experts (smes). previous experiments of our approach for lengthy clinical note summarization using general-purpose models like chatgpt showed their ability to understand clinical terms and abbreviations without finetuning @xcite . this supports our hypothesis that llms can efficiently generate accurate summaries without hallucinations when provided with the relevant contextual data. our method is customizable, contextual, and bypasses the need for extensive training datasets. it harmonizes unstructured clinical data and can be applied to any domain requiring contextualized summaries for various users. ## methods we developed an ai pipeline to automate clinical workflow tasks for generating discharge summaries, leveraging large language models (llms) to answer questions, retrieve information, infer actions, and summarize the information in a stylized format, where a domain expert guided the questions and output style (see figure 2 ). we call our approach "gp-prompting". our first iteration used the general-purpose llm meta-llama-3-8b-instruct @xcite installed locally on a a10g gpu. the following is a summary of our pipeline: ## evaluation outputs for all 10,962 targets were submitted to the shared task and evaluated against a hidden subset of 250 records by the task organizers as described in @xcite . results are presented in time and computational constraints limited our ability to fully optimize the pipeline during the contest. post-contest, we tested a few-shot approach for a single record with five example prompt-data pairs, showing promising results for bhc generation. ## results initial results showed promising performance, though further optimization with few-shot learning and refined model parameters could improve accuracy and efficiency. ## discussion our approach demonstrated a simple and effective method for automatically generating the "brief hospital course" and "discharge instructions" sections of discharge summary notes. future improvements include integrating few-shot learning, fine-tuning, and principled chunking with retrieval-augmented generation (rag). experimentation with various llm sizes and optimization of parameters (e.g., temperature, different values for top_k), topic tracking, and integration of structured chart data (not available for this task) can enhance output quality and speed. ## limitations the task dataset was unrealistic, lacking essential components present of typical charts such as daily progress notes, procedure notes, labs, vitals, microbiology, radiology reports, and medication administration records (mar). generating the bhc and discharge instructions without comprehensive event data leads to hallucinations. the provided dataset, containing only discharge summary notes, is insufficient for accurate bhc or discharge instructions, especially for patients with extended hospital stays. additionally, the target dataset sections were often inaccurately segmented from the input. approximately 16% of phase 2 bhc targets were severely incomplete, often under 100 words. in these cases, the extraction was truncated due to an unexpected heading, often missing the problembased treatment course entirely. the targets also often incorrectly included content from the "transitional issues" section, which should be separate from the bhc. we lacked comprehensive data such as daily progress notes and outpatient referrals, so we utilized selected parts of the discharge summary, including the hpi medication list, which provided partial relevant information needed for the bhc. all selected input sections were considered by the clinician to be accessible during the typical clinical workflow. incomplete records often resulted in outputs lacking the full content of the target data, but it was reassuring that the model did not hallucinate. pipeline challenges: due to data-use agreements, models and data had to be run locally and securely, necessitating downloaded llms. this limitation prevented the use of faster, publicly available pipelines, resulting in lower accuracy, and slower local model outputs compared to more advanced models that we plan to use in the future. we also noted a discrepancy between contest and internal bertscores. at the time of this publication, the root cause of this discrepancy is unknown, but it is likely resulting from using different bertscore functions (we used a standard "bert_score" import, whereas the contest scoring used a custom bertscore script). similarly, the custom alignscore and medcon scores used for contest were not implemented during our evaluation process as we were unable to successfully run the custom scripts in time for the contest entry. ## conclusion the solution we presented was an efficient, context-aware, question-based approach to automate the generation of discharge summaries. despite the constraints and limitations of the dataset and evaluation metrics, our method showed promise, particularly with a few-shot learning approach. future work will focus on refining chunking methods for a rag-based approach, optimizing prompts, and exploring various llm configurations to improve accuracy and reliability in clinical settings.