Y ale at “Discharge Me!”: Evaluating Constrained Generation of Discharge Summaries with Unstructured and Structured Information (2024)

Abstract: In this work, we propose our top-ranking (2nd place) pipeline for the generation of discharge summary subsections as a part of the BioNLP 2024 Shared Task 2: “Discharge Me!”. We evaluate both encoder-decoder and state-of-the-art decoder-only language models on the generation of two key sections of the discharge summary. To evaluate the ability of NLP methods to further alleviate the documentation burden on physicians, we also design a novel pipeline to generate the brief hospital course directly from structured information found in the EHR. Finally, we evaluate a constrained beam search approach to inject external knowledge about relevant patient problems into the text generation process. We find that a BioBART model fine-tuned on a larger fraction of the data without constrained beam search outperforms all other models.

## introduction discharge summaries are vital sources of information that provide a bridge between inpatient treatment and continuation of care in rehabilitation, outpatient, or other intermediate settings. these summaries are often the only form of communication that follows a patient to their next setting of care @xcite . this documentation serves many roles, including next action items necessary for the patient, clear identification of incidental findings necessitating follow-up, new treatment regiments, and many other important components of patient treatment plan @xcite . the discharge summary is a complex document that addresses not only a wide array of members of the care team including the patient's primary care physician, specialists, ancillary departments, but also the patient themselves. within the discharge summary, two sections are particularly instrumental in the continuity of care and complex in their content: the brief hospital course (bhc) and the discharge instructions. the bhc summarizes the course of events that occurred from the moment a patient presents to the emergency department (ed) through their hospital course, ending in discharge. this summary is often structured by problem list or procedure and depends heavily on the discharging service (medical vs. surgical etc.) discharge instructions serve to inform the patient through lay language about key details of their hospital stay, as well as to structure the complex follow-up care that patients will navigate after discharge, enabling them to manage their health effectively in collaboration with their outpatient medical team @xcite . large language models (llms), such as chat-gpt, offer a potential solution to the long-standing issue of inaccessible medical communication and the time-demanding nature of synthesis of discharge summaries. creating high-quality discharge summaries is a challenging and time-consuming task. significant prior work has demonstrated the broad capabilities of llms in clinical natural language processing @xcite @xcite . this work suggests that llms could be leveraged for the automated generation of discharge summaries. the automatic generation of discharge summaries from inpatient documentation could support alleviating the burden of clinical documentation, particularly under the significant pressures of the inpatient setting @xcite . ## limitations while our methods were able to produce reasonable bhcs and discharge instructions, there are several important limitations to our study. computationally, we were unable to perform a rigorous hyperparameter search across all our experimental conditions due to computational constraints. there is potential for improvement given further resources. namely, we were surprised that constrained generation performed significantly worse than vanilla biobart models. this is potentially due to the additional hyperparameters that need to be tuned, including the tokens of interest that medcat identified and beam sizes. furthermore, we believe that there is limited clin-ical validity in the current task as it has been framed. the automated generation of bhc and discharge instructions utilizing physician generated preceding text does not truly automate the task, nor does it obviate the need for the core summarization task of note-writing on the part of the physician (rather than documentation of findings). we attempted to model a more representative use case by including the generation of pseudo-soap notes but found significantly worse performance, demonstrating this difficulty of the real-world clinical task. furthermore, the format and physician-and institutionspecific stylistic choices had a significant impact on automated performance, as demonstrated by the significant variation in documentation length and lack of standard templates even within services that discharged patients. the challenge organizers did attempt to alleviate concerns around generalizability with a qualitative analysis by clinicians, but further efforts in automated metrics involving semantic comparison are necessary. ## conclusion in this work, we present experiments for the automated generation of brief hospital courses and discharge instructions from structured and unstructured data captured during an ed encounter. we find that a biobart model with increased training data performed better than both other encoderdecoder models and a black-box decoder-only model. we also find that constraining generation to emphasize generation of umls concepts worsens performance. finally, we show that gpt-3.5 can generate brief hospital courses purely from structured information, further reducing the annotation burden for physicians.