QUB -Cirdan at “Discharge Me!”: Zero shot discharge letter generation by open-source LLM (2024)

Abstract: The BioNLP ACL’24 Shared Task on Streamlining Discharge Documentation aims to reduce the administrative burden on clinicians by automating the creation of critical sections of patient discharge letters. This paper presents our approach using the Llama3 8B quantized model to generate the “Brief Hospital Course” and “Discharge Instructions” sections. We employ a zero-shot method combined with Retrieval-Augmented Generation (RAG) to produce concise, contextually accurate summaries. Our contributions include the development of a curated template-based approach to ensure reliability and consistency, as well as the integration of RAG for word count prediction. We also describe several unsuccessful experiments to provide insights into our pathway for the competition. Our results demonstrate the effectiveness and efficiency of our approach, achieving high scores across multiple evaluation metrics.

## introduction the bionlp acl'24 shared task, "discharge me!" on codabench @xcite , focuses on automating the creation of two crucial sections of patient discharge letters: "brief hospital course" (bhc) and "discharge instructions" (di). this initiative arises in response to significant time burdens on clinicians, highlighted by surveys of u.s. physicians. one study found that physicians spend twice as much time on electronic health records (ehr) compared to direct patient interactions during clinical hours @xcite . another survey involving 1,524 physicians revealed an average of 1.84 hours spent on ehr documentation outside office hours. automating the generation of bhc and di aims to significantly reduce the clerical load on healthcare providers, thereby improving patient service quality and potentially mitigating clinician burnout. a discharge letter, or a discharge summary, is a critical document summarizing a patient's hospital visit from admission to discharge, serving as a bridge between hospital care and follow-up with outpatient providers. among its several sections, the "brief hospital course" outlines the patient's treatment and progress during the hospital stay, typically using clinical jargon best understood by healthcare professionals. conversely, the "discharge instructions" are designed to guide patients and their caregivers once they leave the hospital, using layman's language to clearly explain follow-up care, medication regimens, and lifestyle recommendations. large language models (llms) offer a promising solution for automating medical documentation due to their ability to understand and generate human-like text @xcite . unlike traditional extractive summarization @xcite , which predominantly involves concatenating snippets from existing texts, llms can enhance summarization by integrating both extractive and abstractive techniques. this has been applied to progress note summarization @xcite , similar to this codabench challenge. with both proprietary llms such as @xcite and open-source llms such as llama3 (ai@meta, 2024), the potential for creating accessible medical summaries is significant. in this challenge, we propose a zero-shot approach utilizing the llama3 8b quantized model, which is optimized for low computing resource usage without fine-tuning, and the result is in the top 10 in the final benchmark assessment. our key contributions are: ## related work the application of foundation models, pre-trained on billions of tokens from diverse data sources, is increasingly prevalent in healthcare @xcite . these models are pivotal in various domains, such as diagnosis generation @xcite and medical image analysis @xcite . within clinical text processing, large language models (llms) are employed for tasks including summarization @xcite and answering medical questions @xcite . specifically, the "discharge me!" challenge involves condensing extensive medical records into succinct discharge letters while retaining all critical information, making llms suited for this task. participants in the bionlp 2023 workshop's problem list summarization task often utilized t5 @xcite or bart @xcite ) models, enhancing these backbones either by further training on clinical texts or fine-tuning for specific clinical tasks @xcite . this further pre-training introduces medical knowledge not originally present in the llm while fine-tuning adapts the model to produce outputs in the correct format for the target task. several studies such as biomistral @xcite and pmc-llama @xcite have adapted open-source llms by applying pre-training and fine-tuning sequentially. conversely, med-palm @xcite bypasses additional pre-training, relying solely on fine-tuning from a vast pre-trained dataset. on a different note, biomedlm @xcite focuses exclusively on medical texts, resulting in a smaller model but still competes effectively with models trained on larger, more general datasets. pre-training and fine-tuning llms require gpus with significant memory capacities (often exceeding 16gb). fine-tuning can take several days, even using parameter-efficient fine tuning (peft) methods like lora @xcite . however, modern llms can exhibit strong performance without additional fine-tuning if provided with the appropriate context and instructions. for instance, almanac @xcite enhances its output by retrieving clinical question-related knowledge from curated sources, a technique known as retrieval-augmented generation (rag) @xcite . additionally, medagents @xcite demonstrates that a zero-shot method, which deconstructs the question into distinct steps and assigns specific prompts and roles to the llm for each stage, can achieve competitive results compared to more traditional few-shot approaches. ## methods in this section, we introduce our zero-shot template-based approach, combined with rag, to determine the target word count, which is both effective and resource-friendly. we adopted the llama3 8b model with 8-bit quantization as the open-source model for this challenge. figure 1 illustrates our approach: while gpt-4/3.5 models generally outperform open-source models such as llama2 in understanding ehr data @xcite , the rules of this challenge discourage the use of proprietary model apis (e.g., openai's gpt-4). consequently, we resorted to the state-of-the-art (sota) open-source model, llama3 (ai@meta, 2024). our approach leverages the full text from the "text" field in the provided discharge.csv file, alongside aggregated fields from other mimic-iv tables, including patient information, diagnoses, and transfer history. we meticulously curated a template for each target section and designed prompts to guide the llm in generating the required sections. in addition to our final approach, we documented several other zero-shot methods for target section generation and various approaches to predict the target section's word count. however, these were not adopted in our final solution. ## results the llama3 model was downloaded from the ollama model repository with the model id "llama3:8b-instruct-q8_0". we utilized the langchain framework for retrieval, template building, and model calling. all experiments were conducted on a t4 gpu with 16gb memory, using the microsoft azure platform's "standard nc4 as t4 v3 (4 vcpus, 28 gib memory)" configuration. we compared several approaches: 1. baseline with random shuffling: we shuffled the "hadm_id" column, a unique identifier for each patient's discharge letter, assigning a random target section to each "hadm_id". this random selection comes from the same distribution as the training data but without the actual content of the input text. ## unsuccessful attempts we also explored several alternative approaches for this task, but they yielded unsatisfactory results: we asked the llm to use the style of the retrieved target section to fit the patient context. however, the llama3 8b model often used the target section directly, failing to infer the style and remove the original content. this could be due to the weaker reasoning ability of the 8b model compared to the 70b model with better reasoning ability. ## conclusion in this paper, we present a resource-friendly approach to automating the generation of the "brief hospital course" and "discharge instructions" sections in discharge letters using the llama3 8b quantized model. our zero-shot template-based method and retrieval-augmented generation produce high-quality, contextually appropriate summaries. however, we observe a lower bleu score due to the different wording between the method's result and the target sections. ensuring the reliability and accuracy of generated content remains a significant challenge. future work will focus on enhancing model reasoning capabilities, improving dynamic template extraction, and integrating robust validation mechanisms to verify medical accuracy. the code for this work is shared on https://github.com/ruiguo-bio/ discharge_me , covering aggregating additional tables, segmentation of the discharge letters, rag for the two target sections, and the random forest classifier for the target section words prediction.