UF - HOBI at “Discharge Me!”: A Hybrid Solution for Discharge Summary Generation Through Prompt-based Tuning of G ator T ron GPT Models (2024)

Abstract: Automatic generation of discharge summaries presents significant challenges due to the length of clinical documentation, the dispersed nature of patient information, and the diverse terminology used in healthcare. This paper presents a hybrid solution for generating discharge summary sections as part of our participation in the “Discharge Me!” Challenge at the BioNLP 2024 Shared Task. We developed a two-stage generation method using both extractive and abstractive techniques, in which we first apply name entity recognition (NER) to extract key clinical concepts, which are then used as input for a prompt-tuning based GatorTronGPT model to generate coherent text for two important sections including “Brief Hospital Course” and “Discharge Instructions”. Our system was ranked 5th in this challenge, achieving an overall score of 0.284. The results demonstrate the effectiveness of our hybrid solution in improving the quality of automated discharge section generation.

## introduction the discharge summary is one of the most crucial documents that capture patients' present illness, diagnostic findings, therapeutic procedures, and follow-up instructions @xcite . timely, high-quality discharge records can remarkably reduce the risk of patient readmissions, ensuring continuous and coordinated patient care, supporting the decision-making process, and bridging the information gap between healthcare providers @xcite @xcite . however, manually writing discharge summaries is time-consuming and errorprone, given the complexity of clinical information, the dispersed nature of patient details, and the increasing burden of clinical documentation @xcite . ## dataset the "discharge me!" @xcite challenge dataset foot_0 is curated from the mimic-iv database @xcite and features over 109,000 ed visits. each record includes icd-9 or icd-10 diagnosis codes, chief complaints, at least one radiology report, and a discharge summary with "brief hospital course" and "discharge instructions". the dataset was split into training (68,785 samples), validation (14,719 samples), phase i testing (14,702 samples), and phase ii testing (10,962 samples) subsets. the phase ii testing dataset will serve as the final test set. all datasets and tables are derived from the mimic-iv submodules. the challenge focuses on the automated generation of the "brief hospital course" and "discharge instructions" sections. ## methods triggered by the recent rag-based summarization methods, we developed a hybrid solution that is composed of a "retriever" and a "generator". we fine-tuned an encoder-only clinical llm, gatortron, as the retriever to identify important discharge summary primary:\ncopd exacerbation\n\ nsecondary:\nafib\ nanxiety\nhtn\ncad 1. acetaminophen 325 mg po q4h:prn pain \ n2. albuterol sulfate 90 mcg/actuation inhalation q4h \... ms. ___ is a ___ female with history of \ncopd on home o2, atrial fibrillation on apixaban, hype... segment by sections selected subset 1 selected subset 2 original form name entity recognition acetaminophentreatment", "pain -mild-problem", "feverproblem", "citalopramtreatment", "cyanocobalamintreatment" ner results concatenate p-tuning gatortrongpt gatortron ms. ___ is a ___ female with history of \ncopd on home o2, atrial fibrillation on apixaban, hype... generated results figure 1: overview of our summary generation pipeline item total count visits 109,168 discharge summaries 109,168 radiology reports 409,359 ed stays 109,403 ed diagnoses 218,376 to reduce the length of the input, we used a rulebased method to segment the notes into individual sections. we manually examined several notes from the training set to identify (1) a subset of sections that are directly related to the target sections, and (2) a subset of sections useful but not directly relevant to the target sections. the input was reconstructed by concatenating: (1) original text from the sections directly related to the target sections, (2) clinical concepts extracted using gatortron from the sections useful but not directly related to the target sections, and (3) diagnosis descriptions. to instruct gatortrongpt to generate target sections, we explored four strategies by combining different tuning methods and input construction methods: (1) traditional fine-tuning using original inputs, (2) fine-tuning using our reconstructed inputs, (3) prompt-based tuning (p-tuning) using original inputs, and (4) p-tuning using our reconstructed inputs. the following sections highlight the approach that demonstrated the best performance-p-tuning using both ner results and original texts. this method integrates several advanced techniques and models to optimize outcomes. as illustrated in figure 1 , our best strategy combines the genera-tive capabilities of state-of-the-art clinical large language models, the extractive ability of ner systems, and efficient instruction using soft-prompt tuning techniques. the experimental results show that our approach can generate coherent contexts for the two important clinical sections. the following subsections describe the models and methods used in our study, including the model architectures, training strategies, and evaluation metrics. ## conclusion this paper presents a hybrid system developed by our team in participating in the "discharge me!" challenge at the bionlp 2024 shared task. we developed a hybrid system by combining extractive and abstractive summarization techniques. our solution is triggered by the retrieval augmented generation (rag) strategy that consists of a retriever to identify relevant information and a generator to generate the content. we fine-tuned gatortron to recognize important problems, treatments, and lab tests from clinical notes as the retriever. we applied a clinical generative llm, gatortrongpt, as the generator to generate the target sections. our approach was ranked 5th place among the participating teams, achieving an overall score of 0.284. using a fine-tuned encoder-only llm, gatortron, as the retriever, our system is able to capture important clinical concepts to reduce the input length and to focus the generator, gatortrongpt, on those important clinical concepts. this strategy alleviated the challenge of token limitation in llms when dealing with clinical documents with extensive lengths. by integrating ner results from selected subset one with original text from the other selected subset, the input size was reduced by approximately 80% on average, which enabled the gatortrongpt model to operate more efficiently and effectively. the human evaluation results offer valuable insights into the quality of the generated discharge summaries. by conducting an average score from three clinicians, we got a more unbiased humanassessed performance of our generation pipeline. this process guides future enhancements to our model and data preprocessing methods. overall, our study demonstrates the effectiveness of a hy-brid approach that leverages both extractive and abstractive techniques in the generation of discharge summaries. the integration of ner and advanced generative modeling not only improves the manageability and performance of the task but also ensures the production of high-quality, contextually appropriate summaries.