I gnition I nnovators at “Discharge Me!”: Chain-of-Thought Instruction Finetuning Large Language Models for Discharge Summaries (2024)

Abstract: This paper presents our proposed approach to the Discharge Me! shared task, collocated with the 23th Workshop on Biomedical Natural Language Processing (BioNLP). In this work, we develop an LLM-based framework for solving the Discharge Summary Documentation (DSD) task, i.e., generating the two critical target sections ‘Brief Hospital Course’ and ‘Discharge Instructions’ in the discharge summary. By streamlining the recent instruction-finetuning process on LLMs, we explore several prompting strategies for optimally adapting LLMs to specific generation task of DSD. Experimental results show that providing a clear output structure, complimented by a set of comprehensive Chain-of-Thoughts (CoT) questions, effectively improves the model’s reasoning capability, and thereby, enhancing the structural correctness and faithfulness of clinical information in the generated text. Source code is available at: https://anonymous.4open.science/r/Discharge_LLM-A233

## introduction discharge summaries encapsulate key details of a patient's hospitalization, from admission to discharge. these documents, however, can contain excessive amount of medical notes, making it difficult for subsequent caregivers or patients to quickly understand essential past medical information. brief hospital course and discharge instructions then become two critical sections in discharge summaries to address this issue. the former outlines critical hospital events for healthcare providers, while the latter offers post-discharge care instructions to patients and their caregivers. the discharge me! shared task 1 @xcite at the bionlp workshop, known as discharge summary documentation (dsd), focuses on efficiently generating these two critical sections, a task that is both challenging and time-consuming for clinicians. ## related work in recent years, large language models (llms) like gpt-2 or gpt-3 have excelled in nlp tasks such as language generation, question answering, due to their vast number of paramters and extensive training on diverse datasets. these models can be adapted to new domains and tasks through methods like prompting, which uses natural language instructions @xcite or few-shot examples @xcite . however, considering dsd problem, the length and excessive information in discharge summaries hinders their use as examples for few-shot prompting. alternatively, parameter-efficient fine-tuning, which freezes an llm weights and inserts a small number of tunable parameters @xcite , has proven effective in specialized clinical tasks like radiology report generation @xcite . from the clinical summarization perspective, research towards th dsd task was very limited. but there has been a growing focus on many clinical text generation tasks, encompassing radiology reports @xcite , clinical notes @xcite , and summary of diagnoses and patient problems @xcite . ## problem description a discharge summary can contain several free-text sections and medical notes compiled from ehr. dsd input pertinent results (radiology report) radiology report text (radiology.csv) discharge summary text (discharge.csv) chief complaint history of present illness 1. section extraction 2. radiology report selection brief hospital course adapter discharge instruction adapter 3. target section generation examination: the type of diagnostic test (e.g., ct scan, mri) or examination performed, on which organs. ## methodology we propose the discharge-llm framework, which adapt llm to the each generation task of dsd, illustrated in figure 1 . discharge-llm applies three steps, namely section extraction, radiology report selection and target section generation to generate the two critical target sections given discharge summary and radiology report information of a patient's hospital visit. note that we utilize the generated bhc for the subsequent generation of di. ## conclusion in this paper, we present a llm-based framework for discharge summary documentation that adopts several prompting strategies into instructionfinetuning an llm, which enhances structural correctness and faithfulness of clinical information in generated target sections. using small and opensource llms, our work also shows the feasiblity of developing and deploying future lightweight nlp systems locally for confidential clinical tasks.