What’s in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarization (2021)

Abstract: Summarization of clinical narratives is a long-standing research problem. Here, we introduce the task of hospital-course summarization. Given the documentation authored throughout a patient’s hospitalization, generate a paragraph that tells the story of the patient admission. We construct an English, text-to-text dataset of 109,000 hospitalizations (2M source notes) and their corresponding summary proxy: the clinician-authored “Brief Hospital Course” paragraph written as part of a discharge note. Exploratory analyses reveal that the BHC paragraphs are highly abstractive with some long extracted fragments; are concise yet comprehensive; differ in style and content organization from the source notes; exhibit minimal lexical cohesion; and represent silver-standard references. Our analysis identifies multiple implications for modeling this complex, multi-document summarization task.

## introduction the electronic health record (ehr) contains critical information for clinicians to assess a patient's medical history (e.g., conditions, laboratory tests, procedures, treatments) and healthcare interactions (e.g., primary care and specialist visits, emergency department visits, and hospitalizations). while medications, labs, and diagnoses are documented through structured data elements and flowsheets, clinical notes contain rich narratives describing the patient's medical condition and interventions. a single hospital visit for a patient with a lengthy hospital stay, or complex illness, can consist of hundreds of notes. at the point of care, clinicians already pressed for time, face a steep challenge of making sense of their patient's documentation and synthesizing it either for their own decision making process or to ensure coordination of care @xcite . automatic summarization has been proposed to support clinicians in multiple scenarios, from making sense of a patient's longitudinal record over long periods of time and multiple interactions with the healthcare system, to synthesizing a specific visit's documentation. here, we focus on hospital-course summarization: faithfully and concisely summarizing the ehr documentation for a patient's specific inpatient visit, from admission to discharge. crucial for continuity of care and patient safety after discharge @xcite , hospital-course summarization also represents an incredibly challenging multi-document summarization task with diverse knowledge requirements. to properly synthesize an admission, one must not only identify relevant problems, but link them to symptoms, procedures, medications, and observations while adhering to temporal, problem-specific constraints. our main contributions are as follows: (1) we introduce the task of hospital-course summarization; ## related works summarization of clinical data and documentation has been explored in a variety of use cases @xcite . for longitudinal records, graphical representations of structured ehr data elements (i.e., diagnosis codes, laboratory test measurements, and medications) have been proposed @xcite . interactive visualizations of clinical problems' salience, whether extracted from notes @xcite or inferred from clinical documentation @xcite have shown promise @xcite . most work in this area, however, has focused on clinical documentation of a fine temporal resolution. traditional text generation techniques have been proposed to synthesize structured data like icu physiological data streams @xcite . @xcite use a transformer model to write ehr notes from the prior 24 hours, while @xcite perform disease-specific summarization from individual progress notes. @xcite develop a distant supervision approach to generate extractive summaries to aid radiologists when interpreting images. @xcite ; @xcite ; sotudeh gharebagh et al. (2020) generate the "impression" section of the radiology report from the more detailed "findings" section. finally, several recent works aim to generate ehr notes from doctor-patient conversations @xcite @xcite . recent work on summarizing hospital admissions focuses on extractive methods @xcite @xcite . ## hospital-course summarization task given the clinical documentation available for a patient hospitalization, our task of interest is to generate a text that synthesizes the hospital course in a faithful and concise fashion. for our analysis, we rely on the "brief hospital course" (bhc), a mandatory section of the discharge note, as a proxy reference. the bhc tells the story of the patient's admission: what was done to the patient during the hospital admission and why, as well as the follow up steps needed to occur post discharge, whenever needed. nevertheless, it is recognized as a challenging and time consuming task for clinicians to write @xcite ; uc irvine residency, 2020). ## dataset analysis & implications to motivate future research in multiple, selfcontained directions, we distill task-specific characteristics to a few salient, standalone takeaways. for each takeaway, we provide evidence in the data and/or literature, before proposing implications of findings on model development and evaluation. background. clinicians copy forward information from previous notes to save time and ensure that each note includes sufficient evidence for billing and insurance purposes @xcite . copy-paste is both widely used (66-90% of clinicians according to a recent literature review @xcite ) and widely applied (a recent study concluded that in a typical note, 18% of the text was manually entered; 46%, copied; and 36% imported foot_0 (wang et al., 2017)). please see appendix d for more information on the issue of copy-paste. analysis -extractiveness. clinsum appears very extractive: a high coverage (0.83 avg / 0.13 std) and a very high density (13.1 avg / 38.0 std) @xcite for a description of the statistics). however, we find that 64% of the extractive fragments are unigrams, and 25% are bigrams, which indicate a high level of re-writing. the density measure is large because the remaining 11% of extractive fragments are very long. yet, there is a strong positional bias within summaries for long fragments. figure 1 , groups fragments according to their relative order within each summary. the longest fragments are usually first. qualitative analysis confirms that the beginning of the bhc is typically copied from a previous note and conveys the "one-liner" (e.g., pt is a 50yo male with history of chf who presents with edema.) this abrupt shift in extractiveness should affect content selection. in particular, when look- in figure 2 , two supporting trends emerge. (1) on average, one sentence accounts for roughly 50% 3 of the overall r 12 score. (2) afterwards, the marginal contribution of the next shrinks, as well as the r 12 gap between the best sentence and the minimum / average, according to the oracle. there should also be evidence of the copy-paste positional bias impacting content selection. unsurprisingly, a model (clinneusum) trained on oracle gain extractions gets progressively worse at mimicking it. specifically, for each extractive step, there exists a ground-truth ranking of candidate sentences by relative r 12 gain. as the relevance gap between source sentences shrinks (from figure 2 ), clinneusum's predictions deviate further from the oracle rank ( ## conclusion based on a comprehensive analysis of clinical notes, we identify a set of implications for hospitalcourse summarization on future research. for modeling, we motivate (1) the need for dynamic hybrid extraction-abstraction strategies (4.1); (2) retrievalaugmented generation (4.3); and (3) the development of heuristics to assess reference quality (4.5). for evaluation, we argue for (1) methods to assess factuality and discourse which are associative in nature, i.e., incorporate the complex inter-dependence of problems, medications, and labs (4.2, 4.4); and