e-Health CSIRO at “Discharge Me!” 2024: Generating Discharge Summary Sections with Fine-tuned Language Models (2024)

Abstract: Clinical documentation is an important aspect of clinicians’ daily work and often demands a significant amount of time. The BioNLP 2024 Shared Task on Streamlining Discharge Documentation (Discharge Me!) aims to alleviate this documentation burden by automatically generating discharge summary sections, including brief hospital course and discharge instruction, which are often time-consuming to synthesize and write manually. We approach the generation task by fine-tuning multiple open-sourced language models (LMs), including both decoder-only and encoder-decoder LMs, with various configurations on input context. We also examine different setups for decoding algorithms, model ensembling or merging, and model specialization. Our results show that conditioning on the content of discharge summary prior to the target sections is effective for the generation task. Furthermore, we find that smaller encoder-decoder LMs can work as well or even slightly better than larger decoder-based LMs fine-tuned through LoRA. The model checkpoints from our team (aehrc) are openly available.

## introduction clinical documentation in the age of electronic health records (ehrs) can be a significant burden to clinicians in recording clinical information effectively @xcite . this reduces the time clinicians spend interacting with their patients and could lead to stress and burnout @xcite , degrading both the quality of patient care and the experience of care providers @xcite . language models (lms) have demonstrated impressive nlp capabilities and are considered to have the potential to reduce the clinical documentation burden by automatically generating clinical text @xcite @xcite . for example, a recent study @xcite demonstrated that lms can generate succinct clinical summaries from text including progress notes and patient-doctor dialogues, sometimes even preferred over those written by medical experts. the bionlp 2024 shared task "discharge me!" @xcite focuses on generating the discharge summary (or discharge note) to assess the potential of lms for this specific type of clinical note, which is often more time-consuming for clinicians to document and also more challenging to model given its length and complexity. this paper presents the submissions from e-health csiro in the shared task. we approach the task by fine-tuning multiple open-sourced lms, including both decoder-only and encoder-decoder models. we fine-tune these models to generate two specific sections from discharge notes: brief hospital course and discharge instruction, by conditioning on the prior content in the notes as context. we explore various configurations with input context, decoding, ensembling, and target specialization. we find that much smaller encoder-decoder lms could have a slight edge over fine-tuning decoderonly lms (all with the size of 7/8b parameters) with lora @xcite . our best submission ranked 3 rd on the final leaderboard under both automatic and manual evaluation. ## discussions we demonstrate that fine-tuning lms based solely on the prior content from the discharge note is sufficient to generate bhc and di sections. given the heterogeneity of ehr data @xcite and variations in clinical notes @xcite , selecting the appropriate inputs would be crucial for both the quality and applicability of the generation. in this work, we assume that the non-bhc/di contents of the discharge note have been populated from other available sources or clinical notes, making them readily available as model input. the context for bhc ("sections part 1" in figure 1 ) typically includes chief complaint, history of present illness, past medical history, social history, physical exam, and various pertinent results. the "sections part 2" of di context may include admission and discharge medications, discharge disposition, dischage diagnoses. using these sections as input yields competitive generation results, and including additional text sources like radiology reports does not lead to improvement. one explanation is that the sections within the discharge summary, such as "pertinent results", often already include imaging findings. future work may futher investigate how selecting from the modeling perspective, we find that finetuning smaller lms, such as primera, achieves surprisingly good results. examination of any potential biases or overfitting is left for future work. during development, we observed that the generation qualities of llama3 and primera were similar (examples shown in appendix given the slight edge over llama3 and other lms, primera was our final submission. similar to previous findings @xcite , we see that the manual evaluation aligns with the automatic evaluation in ranking different systems. the manual evaluation further reports fine-grained scores on completeness, correctness, and readbility for bhc and di separately. interestingly, we observe that primera obtains the best overall score for di but worst for bhc among the top-5 teams. this may indicate the model capacity correlates with the length or complexity of the target generation, with smaller lms potentially struggling with prolonged outputs. it is plausible that llama3 would offer improved results on bhc, especially in terms of readability. future work may investigate this further through separate automatic evaluations specifically for bhc and di. ## conclusion this paper describes our efforts in the "discharge me!" bionlp 2024 shared task @xcite , with the final system ranked 3 rd on both automatic and manual evaluation. we show that fine-tuning lms with appropriate input context has the potential to automatically synthesize high-quality discharge summary sections, which holds promise to reduce the time clinicians spend on documentation.