EPFL - MAKE at “Discharge Me!”: An LLM System for Automatically Generating Discharge Summaries of Clinical Electronic Health Record (2024)

Abstract: This paper presents our contribution to the Streamlining Discharge Documentation shared task organized as part of the ACL’24 workshop. We propose MEDISCHARGE (Meditron-7B Based Medical Summary Generation System for Discharge Me), an LLM-based system to generate Brief Hospital Course and Discharge Instruction summaries based on a patient’s Electronic Health Record. Our system is build on a Meditron-7B with context window extension, ensuring the system can handle cases of variable lengths with high quality. When the length of the input exceeds the system input limitation, we use a dynamic information selection framework to automatically extract important sections from the full discharge text. Then, extracted sections are removed in increasing order of importance until the input length requirement is met. We demonstrate our approach outperforms tripling the size of the context window of the model. Our system obtains a 0.289 overall score in the leaderboard, an improvement of 183% compared to the baseline, and a ROUGE-1 score of 0.444, achieving a second place performance in the shared task.

## introduction in modern healthcare, the electronic health record (ehr) is a fundamental part of clinical practices as it ensures the documentation of a patient's medical journey. essential to this record are the clinical notes seriously crafted by physicians postconsultation. these notes encapsulate crucial details ranging from the patient's reason for the visit to their medical history, symptoms, diagnosis, and recommended treatment plan @xcite . acting as vital components within the ehr, clinical notes foster effective communication among healthcare providers, offer legal protection, and ensure continuity of care @xcite . however, despite their important role, clinical notes impose a substantial time burden for physi-cians. recent research in the u.s. has revealed that physicians spend an average of 1.77 hours daily on documentation tasks outside of consultation hours @xcite . this extensive time investment contributes to pressing healthcare issues such as clinician burnout, excessive workloads, and understaffing @xcite . one area where clinicians encounter notable time constraints is in the creation of discharge summaries and hospital course summaries. crafting these summaries to be both concise and comprehensive demands considerable effort. to address this challenge, there is a pressing need to streamline the summary generation of these sections. people try to use machine learning to automate these summaries, but all face the difficulty of models with limited abilities, domain-specific terminologies, and reasoning over specialized knowledge @xcite @xcite . bionlp acl'24 shared task on streamlining discharge documentation focuses on solving the summarization challenges. in this competition, participants worked with a dataset derived from mimic-iv, covering 109,168 emergency department (ed) visits. each patient visit record encompasses several key components: the chief complaints logged by the ed, diagnosis codes (either icd-9 or icd-10), at least one radiology report, and a comprehensive discharge summary. the discharge summary includes vital sections "brief hospital course (bhc)" and "discharge instruction (di)". the main objective of this competition is to automate the generation of these two essential sections of the discharge summary @xcite . to solve this shared task, we propose medis-charge, a fully automatic system based on meditron-7b @xcite with context window extension for generating bhc and di sections according to the patient's ehr. the model with a longer context window size helps our system process the full text of most long-context cases. next, we propose a dynamic information selection framework that can improve the robustness of the system since it can prune ehrs with very long context to fit a limited context window size. we conduct a comprehensive evaluation of our system on the full phase ii test set. in the competition, our system obtained an overall score of 0.289 on a held-out subset of this test set, improving over the official baseline (0.102) by 183% relatively. we make our code available at https: //github.com/haotian89/medischarge . with the development of natural language processing (nlp), the automation of clinical documentation has gradually received attention due to its huge application value. at early stages, rule-based nlp approaches have been employed to extract specific information from free-text clinical notes and populate structure fields within the ehr @xcite . machine learning and deep learning techniques such as long short-term memory (lstm) @xcite and transformers @xcite have shown promise in generating simple clinical summaries @xcite . the appearance of large language models (llms) has brought unprecedented changes @xcite , and demonstrated potential strong capabilities in clinical text summarization @xcite . ## the medischarge system our proposed medischarge system is an llmbased system for the automatic generation of discharge summary sections from relevant key components of clinical ehrs (see fig. 2 ). our system consists of three parts: (1) section extraction, (2) instruction fine-tuning medical llm, and (3) robust inference. we aim to utilize llms and refinement techniques to create summaries that ensure factual accuracy in alignment with ehrs and preserve the textual style of clinicians. ## qualitative analysis in addition to quantitative analysis based on different metrics, we also review specific cases manually to intuitively assess whether the generated summaries are of high quality and could potentially aid or substitute clinicians in real-life scenarios. here is one di generated example, and more example cases are in the appendix f. ## conclusion the research presented in this paper highlights the significant advancements made by medis-charge system in the field of automated discharge summary generation at acl'24 bionlp shared task on streamlining discharge documentation. the experiment results demonstrate that our system with efficient information usage and good costs management achieves a great performance improvement of 183% compared to the baseline, and is able to efficiently generate concise and medically accurate discharge summaries, markedly reducing the burden on healthcare professionals. the adoption of an llm, specifically pretrained for medical data, can better complete medical summarization tasks than a general fundamental llm. furthermore, the dynamic information selection framework we proposed shows a robust task inference ability, significantly outperforming the simple truncation strategy and even dominating the context window extension method across several nlp metrics.