Hospital Discharge Summarization Data Provenance (2023)

Abstract: Summarization of medical notes has been studied for decades with hospital discharge summaries garnering recent interest in the research community. While methods for summarizing these notes have been the focus, there has been little work in understanding the feasibility of this task. We believe this effort is warranted given the notes’ length and complexity, and that they are often riddled with poorly formatted structured data and redundancy in copy and pasted text. In this work, we investigate the feasibility of the summarization task by finding the origin, or data provenance, of the discharge summary’s source text. As a motivation to understanding the data challenges of the summarization task, we present DSProv, a new dataset of 51 hospital admissions annotated by clinical informatics physicians. The dataset is analyzed for semantics and the extent of copied text from human authored electronic health record (EHR) notes. We also present a novel unsupervised method of matching notes used in discharge summaries, and release our annotation dataset1 and source code to the community.

## introduction a discharge summary is written by physicians when a patient is discharged from the hospital and provides clinicians with an overview of the patient's hospital stay. physicians often reference or copy ehr notes to the discharge summary. these copied notes include progress notes, consult notes, and test results. we call these previously written medical documents note antecedents since they are written prior to the discharge summary and can be used as source text as input to an automatic summarization (as) system. the flow of information from the note antecedent to the discharge summary is traced by annotating semantically similar content and copied text across notes and sections (such as brief hospital course), which can be copied or paraphrased text. these annotations, called note matches, "tie" the notes' discrete lexical spans of text. each annotation is a text span in the discharge summary that carries a similar or identical meaning to its matching note antecedent as a span, which is a clinical medical note containing the original information that contributed to the discharge summary. in many cases, the text annotated by the note antecedents is later paraphrased or copied into the discharge summary. figure 1a shows a note match annotation of a physician note and the linked text in the discharge summary given in figure 1b . in this example, the patient's encounter is documented in the history of present illness section of the note antecedent and later paraphrased in the summary at the time of the patient's discharge. the red arrow represents the connection between these two lexical spans within an admission. this connection always starts from the original text in any clinical note and terminates in the discharge summary for the respective admission. for this work, both ends of the link are text spans written by physicians (see section 3 for more detail on the annotation process). understanding the extent of meaningful data as opposed to low quality copied structured data is key in determining the feasibility and choice of methods needed to generate discharge summaries. in many instances, the copied text comes from high quality sources such as the history of present illness section of the physician note. however, redundancy, errors and data incoherence is pervasive in the vast amounts of medical data taken from patients during a hospital stay @xcite . as much as 46% of the discharge summary is copied and pasted, 36% imported from structured data sources, leaving only 18% manually entered @xcite . the extent of copied text in notes is well known, but the origins of discharge summary text is not. since it is unclear how concepts arrive into the summary, extrapolating anything more than the measure of copied text and their semantics would necessitate understanding the decisions made by the physician while writing the summary. however, we can infer what is missing from a summarization by subtracting the portion represented by the note antecedents. knowing what is missing from the summary and unique to the physician's direct individual experience is fundamental for gauging the summarization performance upper bound. for this reason, our primary goal for this work is to find and analyze the flow of data from notes written prior to the discharge summary, as shown in figure 1 . it is our position that a better understanding of the provenance of data, by note and section, is a crucial first step before as can be successfully applied to discharge summarization future work. a secondary goal is to produce an unsupervised baseline and model for the research community. to accomplish these goals three clinical physicians annotated admission records from the freely available medical information mart for intensive care iii (mimic-iii) version 1.4 (johnson et al., 2016) corpus foot_0 . these annotations uncovered where notes overlap and offer high quality human examples for supervised learning methods. our contributions are the annotated dataset (see section 3), its analysis (see section 3.2), and a novel unsupervised method using the word mover algorithm @xcite with clustering to assist in the annotation process (see section 4). ## related work most clinical text data analysis has been in the medical field (rather than nlp) with early work in the discovery of data flow using visual summarization methods for clinical psychiatric data @xcite . redundant text detection in samecategory ehr notes using latent dirichlet allocation @xcite has been explored @xcite . duplication in medical ehr notes were studied in recent work to find root causes of copied text using 10-gram token spans, which found that 58% of a physician note is copied from previous notes @xcite . recent examples of cross discipline collaboration include exploring language and terminology differences between nurses and physicians @xcite and patient centric summarization @xcite . soap (subjective, objective, assessment and plan) note generation (a type of physician note) is another example of an area of mutual interest between the nlp and medical domains @xcite . provenance of data as it relates to the task of summarization has included multi-level recurrent neural networks (rnns) used for extractive summarization @xcite . soon after, bart @xcite was used to abstractively condense and smooth the summaries. a large corpus was analyzed and a hybrid extractive/abstractive neural method was used to summarize the brief hospital course section @xcite . the summarization of mimic-iii physician notes is perhaps the most interesting comparison and potentially most impactful @xcite . clinical notes were summarized by fine-tuning the t5 @xcite and bart @xcite ) state of the art seq2seq models and evaluated using the bertscore @xcite ) and rouge @xcite scoring methods. in addition, co-occurring extracted unified medical language system (umls) medical concepts @xcite were also reported. bodenreider concludes with their intention of providing a foundation for future summarization work and acknowledgment of the challenges facing the clinical note summarization, which echoes the motivation of this work. three attending physicians annotated 51 admissions from the mimic-iii version 1.4 @xcite corpus for overlap with the ehr note antecedents. the overlap was annotated by selecting the semantically similar portions of text from the discharge summary as shown in figure 1 . this provides statistics of overlap both at a note and a section level. to find the section overlap, we use the annotated medsecid @xcite corpus for most notes. in the few cases where note section annotations were not available, we used the pretrained medsecid model to automatically section notes. such sections are helpful to medical clinicians when finding topical information and useful for billing insurance companies. we direct the reader to landes et al. for further reading on the function and utility of sections in medical notes. the annotation selection criteria by admission was the note category and the highest note count from the medsecid corpus. more specifically, selection s c for admission a was chosen as s @xmath0, where c is the mimic-iii corpus and |a| are the number of notes in the admission. our early findings showed very little overlap of data with the exception of consult (documentation by consulting physicians across departments) and physician (typically written daily by the physician) notes. for this reason, admissions with these notes took priority in our annotation process to maximize note match overlap. while it could be argued that these notes' statistics may be combined given their likeness in purpose, it was decided to keep them separate for summarization future work. admissions were annotated with note matches using the following process: each note antecedent may have several note matches (as can a discharge summary), but each has a single link across both note documents. an annotation guide was created by the lead physician annotator, who then trained the other two physician annotators. agreement on the criteria for note matches was decided on after each annotator completed one admission. the first admission annotations were then updated per the consensus agreed upon by all annotators. the same process of annotation, discovery, and agreement repeated for three additional admissions; one for each annotator. the remaining 39 were then split among the three annotators. the 51 admission count might give the mistaken impression of a small dataset. however, as shown in ## methods automatic methods to assist in bootstrapping the corpus were considered given the large amount of data the physician needs to sift through to find note matches foot_1 . however, the task is challenging given the pages long document lengths, and precludes transformer pre-training methods. while the task has much in common with paraphrase matching and information retrieval tasks, it is fundamentally different in the way sections of text are matched. for example in question/answer systems, a query matches to an answer in the source text. however, our task requires the correct text span in both note documents. both the discharge summary and the note antecedent predicted text spans for the respective linked note matches are used when evaluating. ## results and discussion the dsprov dataset was provided to the unsupervised algorithm described in section 4 and evaluated against the human annotated note matches. the match spans with the highest flow (see section 4.3) were compared and scored using the measures listed in section 4.1. before the evaluation, bayesian hyperparameter optimization @xcite was used on the human annotated dataset on a subset of the data. the model's hyperparameters were set to the bayesian optimized values and evaluated. of the 569 note matches annotated, an additional 359 were optimized on 500 iterations. this process was repeated on each word embedding for each note match. while the hybrid method explained in section 4.3 had a relatively high semeval partial recall of 69.06 for matching discharge summaries spans, it suffered a low precision score. this implies finding spans is not an issue, but finding correct span boundaries as more difficult. we report both the good recall but poor precision to help explain the kinds of challenges in matching spans between discharge summaries with note antecedents. the performance for the note antecedents matching as shown as mentioned in section 3.1, only icu notes and discharge summaries are provided in mimic-iii. this has the effect of decreasing available information to potentially summarize and also has a negative impact on results as there are fewer examples to match between note documents. however, it has an even greater impact on summarization since the machine cannot generate what is not available in the ehr. while much of this data is available as structured data that can be textually formatted, additional data provenance based methods are needed to use these information sources. for example, it is feasible to summarize sections from structured data such as tabular prescribed medicine to generate the medication history section, which we leave as future work. ## conclusions and future work we have presented dsprov, a new freely available dataset of 569 textual span matches between discharge summaries and note antecedents annotated by clinical informatics physicians. the analysis of our dataset presents new qualitative and quantitative findings of ehr notes. we have also presented a novel unsupervised method for annotating note matches using models tuned on human examples from our dataset. we believe our baseline results and our dataset analysis provide insights necessary for assessing the feasibility of traceable and faithful automatic discharge summarization in future work. while the purpose of this work was to investigate the provenance of discharge summaries from ehr note antecedents, our secondary goal was to augment our dataset with semi-supervised annotations. given the unsupervised baseline models have room for improvement, we leave supervised methods (such as pretrained biomedical language models fine-tuned span tagging) as future work.