Summarizing Patients’ Problems from Hospital Progress Notes Using Pre-trained Sequence-to-Sequence Models (2022)

Abstract: Automatically summarizing patients’ main problems from daily progress notes using natural language processing methods helps to battle against information and cognitive overload in hospital settings and potentially assists providers with computerized diagnostic decision support. Problem list summarization requires a model to understand, abstract, and generate clinical documentation. In this work, we propose a new NLP task that aims to generate a list of problems in a patient’s daily care plan using input from the provider’s progress notes during hospitalization. We investigate the performance of T5 and BART, two state-of-the-art seq2seq transformer architectures, in solving this problem. We provide a corpus built on top of progress notes from publicly available electronic health record progress notes in the Medical Information Mart for Intensive Care (MIMIC)-III. T5 and BART are trained on general domain text, and we experiment with a data augmentation method and a domain adaptation pre-training method to increase exposure to medical vocabulary and knowledge. Evaluation methods include ROUGE, BERTScore, cosine similarity on sentence embedding, and F-score on medical concepts. Results show that T5 with domain adaptive pre-training achieves significant performance gains compared to a rule-based system and general domain pre-trained language models, indicating a promising direction for tackling the problem summarization task.

## introduction the progress note is a common note type in the electronic health record (ehr) that also contains the necessary details for medical billing; therefore, every hospital day will contain at least one progress note for a patient. healthcare providers write them to document a patient's daily progress and care plan @xcite . the progress note contains both subjective and objective information gathered by the care team, and it is updated daily and serves as the most viewed clinical document by providers. the complexity of the progress note increases as the patient's illness worsens with progress notes collected in the intensive care unit (icu) representing the sickest patients in the hospital. in the icu, information and cognitive overload occur frequently, with more opportunities for missed diagnoses and medical errors @xcite . automatically generating a set of diagnoses/problems in a progress note may assist providers in overcoming cognitive biases and heuristics and apply evidence-based medicine via information synthesis to accurately understand a patient's condition. these processes may ultimately reduce the effort in document review and augment care during a time-sensitive hospital event @xcite . clinical note summarization using natural language processing (nlp) has demonstrated promise in previous work. @xcite introduced harvest, an ehr summarizer that is currently deployed at point-of-care in a new york hospital. the nlp components of harvest include a markov chain named-entity tagger that identifies diseases explicitly mentioned in clinical notes and a tf-idf scorer that weighs the importance of the mentions @xcite . with the advances of neural methods, recent work has focused on radiology report summarization @xcite @xcite with pointer generator network @xcite , and doctor-patient conversation summarization @xcite with transformer architectures @xcite . few investigations apply transformers to problem summarization progress notes to identify and generate the top diagnoses during a patient's hospitalization. problem summarization requires complex cognitive processes to arrive at an accurate diagnosis. when a patient is admitted to the hospital, medical evaluations and diagnostics are initially performed to understand a patient's condition. the review is accompanied by documentation in the progress notes to include pertinent details about the patient's symptoms, medications, physical exam findings, radiology findings, laboratory results, etc. these data are organized in the progress note and used with the physician's medical knowledge to arrive at an assessment of the current problems followed by a treatment plan. the system of nonanalytic and analytic reasoning strategies represent clinical diagnostic reasoning, a process involving clinical evidence acquisition with integration and abstraction over medical knowledge to synthesize a conclusion in the form of a diagnosis @xcite . we hypothesize that to summarize a patient's problems and ultimately develop computerized diagnostic decision support systems, the ability of clinical diagnostic reasoning is the key for nlp systems, a gap in existing nlp literature. in this work, we propose a new summarization task designed to meet the real-world need in the hospital setting as the first step to developing nlp models for clinical diagnostic reasoning. the task is built on a new annotation subset of mimic-iii @xcite , a large and publicly available ehr. we formulate the task as a problem list summarization, as we see the task as the first step in a bigger vision of generating entire notes or sections of notes. ultimately, the task is designed with our clinical informatics partners to move toward a future real-world application, where a system generating relevant diagnoses can assist healthcare providers and overcome the cognitive burden and information overload. our contributions include: ## related work in this section, we provide a brief overview of recently published papers on clinical summarization that use neural methods. task setup the stream of recent work on clinical summarization may be divided into two groups: extractive summarization and abstractive summarization. the data corpora are heterogeneous, with multiple note types represented. for extractive sumarization, liang et al. (2019) proposes a summarization task that extracts sentences from progress notes. adams et al.(2021) introduces a clinical note summarization task to generate a discharge summary generated from prior notes during hospitalization. more efforts have been made toward abstractive summarization. several work focus on summarizing radiology reports into an impression, a short piece of text stating the findings from the source image (zhang et al., 2018; macavaney et al., 2019; gharebagh et al., 2020). another task is doctor-patient conversation summarization where the output is a summary describing the patient's visit: (yim and yetisgen-yildiz, 2021; manas et al., input assessment and subjective sections (assessment) pt is a 78 y.o female with h.o copd, htn, recent mva with r.ankle/foot fx who presents with hypoxia and lll infiltrate. chief complaint: pt does not feel better than at admission, still very fatigued and weak. sob unchanged. no chest pain. no other complaints. allergies: no known drug allergies review of systems is unchanged from admission except as noted below review of systems: 2021; @xcite ; or generating clinical notes using both extractive and abstractive summarization: @xcite . our work is similar to @xcite in the emphasis on summarizing problems from progress notes. yet, @xcite uses a disease-specific dataset (hypotension and diabetes), and formulates the problem as extractive summarization. our annotations span a broad range of diagnoses across multiple disciplines (surgery, medicine, neurology, cardiology, trauma, etc.) and investigate extractive and abstractive approaches in the task. evaluation prior work has relied on rouge @xcite as the primary evaluation metric for summarization. most papers also report human evaluation with aspects of clinical relevancy, factual accuracy and readability @xcite @xcite @xcite . few have evaluated using a concept f-score, measuring if the predicted summaries contain accurate medical concepts @xcite . our work follows prior work and uses rouge, concept f-score, and human evaluation to assess the quality of generated summaries. we also evaluate content quality based on semantic representation using bertscore @xcite and cosine similarity for sentence embedding. many clinical nlp applications aim to improve physicians' efficiency and decision-making by automatically highlighting essential information from the large body of textual data in the ehr. the goal of problem summarization is to identify and generate the problems and diagnoses for the patient's icu stay. the problem summarization task could be developed using a multi-document approach with all notes captured during a hospital encounter. a patient encounter may generate multiple clinical notes (e.g. admission note, transfer note, daily progress notes, etc.), involving different modalities of data such as structured ehr data and radiology images. however, we are particularly interested in facilitating nlp model development for clinical diagnostic reasoning. we define the task as single-document summarization and focus only on a cross-sectional point in time with a single progress note. our work will show that summarizing a patient's problems over a single progress note is a challenging task and a necessary foundation that requires clinical text understanding and reasoning over sequences of medical concepts. the progress note is organized in the ubiquitous soap format with four components: subjective, objective, assessment, and plan, a documentation method designed to present patient's problems in a highly structured way and developed by larry weed, md @xcite . each component has multiple sections gathering patients' information, helping the healthcare providers quickly recognize medical events and active problems. subjective sections are written in natural language and record information about health concerns expressed by patients (e.g. chief complaints), and past medical events and history (e.g. allergies, fam-ily history). objective sections are primarily structured data, including vital signs, lab tests, medications. assessment is a brief description of passive and active diagnoses. it states why the patient is admitted to the hospital and the active problem for the day, usually accompanied by the patient's comorbidities. plan section includes multiple subsections, each listing a medical problem and treatment plan. the progress note is timesensitive ehr data because it is documented daily. as a patient's condition changes and the length of stay increases, the progress note may also increase in length. another reason for the increasing size is from copy-and-paste behaviour, also known as "note bloat" adding redundant information or noise and hindering the efficiency in data synthesis, which increases the risk for medical error @xcite @xcite . this reiterates our motivation to develop an nlp system that automatically generates problems and diagnoses to assist providers in clinical workflow and improve diagnostic accuracy. our task took subjective and assessment sections in progress notes as input and omitted the objective sections. both the subjective sections and the assessment section contained information about the reason for admission; therefore, they became the source text (see figure 2 for an example). the reference summary is a list of problems mentioned in each plan subsection relevant to the reasons for hospitalization. we will explain the annotation process in the next section. foot_0 4 data all progress notes were sourced from mimic-iii, a publicly available dataset of de-identified ehr data from approximately 60,000 hospital icu admissions at beth isreal deaconess medical center in boston, massachusetts. we randomly sampled a subset of 768 progress notes and annotated the text spans for the soap components. the goal of the annotation was to obtain lists of problems from the plan subsections. for each plan subsection, the annotators marked the text span for the problem, separating the diagnosis/problems from the treatment or action plans. the annotators subsequently determined if the problem was a primary diagnosis (direct), or a past medical problem or consequence from the primary diagnosis (indirect). two more labels were available for annotating the plan subsection: neither if the problem or diagnosis was not mentioned in the progress note; not relevant if the plan subsection contained non-diagnostic comments such as describing nutrition, prophylaxis, or disposition. finally, we concatenated the direct and indirect problems using semi-colons and used them as reference summary. two medical school students were trained as annotators under the supervision of two board-certified critical care icu physicians. on the four labels, they achieved a cohen's kappa of 0.74 on 10 randomly sampled notes, considered as good quality given the complexity of the task. more details may be found in the appendix a. 2 figure 3 illustrates the task setup. the direct and indirect problems were labeled from each plan subsection using information presented in the input assessment (entire progress note was also available to the annotators for more information), forming the reference summary (all problems in the bottom). a total of 1404 and 1599 text spans were labeled as direct and indirect problems, respectively. the majority of the direct problems were found in the input assessment but many of the indirect problems were not explicitly mentioned in the input assessment and may be found in other parts of the progress note (abdominal pain finding in subjective or pneumothorax finding in chest imaging result of objective). we also performed medical concept mapping through umls (see §5) on the input assessment and kept the overlap with the reference summaries and categorized them explicit mention of problems as an automated labeling approach and baseline. therefore, the problems represent extractive and abstractive medical concepts. we presented the results across these subgroups assuming the complexity increases as we move from explicit to direct to indirect problems. ## experiment setting the unified medical language system (umls) from the national library of medicine is the largest resource containing biomedical concepts and their relationships @xcite . we applied the concept extractor from quickumls (soldaini and goharian), a fast and lightweight python package, to identify all the medical concepts in the text as the baseline system. two state-of-the-art seq2seq transformers were selected to compare with the rule-based method: t5 @xcite and bart @xcite . the transformer models are known as data hungry and pre-trained on domain general text, yet our training data was limited in size but full of medical terms. to help the model learn the medical vocabulary and knowledge, we used data augmentation to generate more training samples for our experiments ( §5.1 and §5.2). ## results and analysis we evaluated all systems on a test set of 92 progress notes and summaries. recall that the progress notes contained the subjective, objective, assessment and plan sections. we set two types of input to the models: (1) assessment section only (assmt), (2) assessment and subjective sections (length permitting) (a+subj). both input settings also had augmented samples from the data augmentation method introduced earlier. we started with a simple rule-based system that was a umls concept extractor on the assessment section. the evaluation metrics across the rule-based, fine-tuned t5 and bart ( §6.1), and t5 with domain adaptation pre-training (dapt, §6.2) are shown in tables 2 and 3 . t5 with dapt outperformed all other systems and established a benchmark performance for the task. we include a qualitative analysis to provide data-driven insights of the task ( §6.3). ## discussion our work begins with a single note in crosssectional design to build our models; however, a patient's hospitalization is a multi-document workflow with repeated measures of progress notes and other note types across several days and multiple providers. in addition, providers generate their diagnoses via a reasoning process that includes structured data from vital signs, laboratory results, etc. images and radiology reports are another modality that highlights the multi-modality approach in diagnostic reasoning. nonetheless, our work opens the door for future research on knowledge intensive clinical summarization. this section includes a discussion of future directions in solving this task. exploring structured data the objective section of the progress note contain embedded structured data, delivering rich information regarding patient's problem. recall the example in figure 3 , the reference summary contains diagnosis: "leukocytosis" (high white blood cell count), "anemia"(low red blood cell count). these diagnosis are usually found in laboratory results. to investigate the use of objective sections and structured data, we append both subjective and objective sections in chronological order to the assessment and input to t5 for fine-tuning and evaluation (t5-all), and we let the t5 tokenizer truncate the text when it exceeds the 512 token limit. on test set, the scores are too low to report. yet, we observed that t5-all, instead of generating medical concepts, often extracts lines of lab values that strongly associate with the disease in reference summary (see figure 7 .1 and 7.2). this preliminary result indicates the future direction of understanding the association between disease and lab values in summarization. ## conclusion we propose a problem summarization task that address diagnostic reasoning, and show that t5 with dapt achieves benchmarking performance for the task, but some key challenges remained. our work lays the ground for future research on knowledge fused clinical summarizers as well as real-world clinical diagnostic decision support system. future work will investigate the uses of structured data, evidence-based evaluation metric and better models for knowledge representation and summarization.