Generation of Patient After-Visit Summaries to Support Physicians (2022)

Abstract: An after-visit summary (AVS) is a summary note given to patients after their clinical visit. It recaps what happened during their clinical visit and guides patients’ disease self-management. Studies have shown that a majority of patients found after-visit summaries useful. However, many physicians face excessive workloads and do not have time to write clear and informative summaries. In this paper, we study the problem of automatic generation of after-visit summaries and examine whether those summaries can convey the gist of clinical visits. We report our findings on a new clinical dataset that contains a large number of electronic health record (EHR) notes and their associated summaries. Our results suggest that generation of lay language after-visit summaries remains a challenging task. Crucially, we introduce a feedback mechanism that alerts physicians when an automatic summary fails to capture the important details of the clinical notes or when it contains hallucinated facts that are potentially detrimental to the summary quality. Automatic and human evaluation demonstrates the effectiveness of our approach in providing writing feedback and supporting physicians.

## introduction studies have shown that the majority of patients do not understand their clinical visits (o' @xcite . after-visit summary note (avs) is a summary given to patients after their clinical visit, it is intended to summarize patients' clinical visits and help their disease self-management @xcite . compared to clinical notes, an aftervisit summary has the following characteristics: 1) it is written in lay person language thus is easy for patients to read and comprehend; 2) it only contains information that patients should be aware of, leaving out redundant details, e.g. unimportant 1 our project page is available at: https://github.com/ pengshancai/avs _ gen lab results, etc. studies have shown that around 36% of american adults have limited health literacy @xcite , and 94.4% of patients found that lay language after-visit summary helps them understand their clinical visits @xcite . however, the implementation of after-visit summary is challenging. many physicians face excessive workloads @xcite and do not have time to complete the summaries in a timely manner @xcite . thus, there is a real need for-and this study contributes to-automatic generation of after-visit summaries to unburdening physicians with complex information workflows. we explore best-performing neural abstractive summarizers to generate after-visit summaries from ehr notes. the summaries are rated by physicians as concise and easy to read. however, they can not be presented directly to patients, as they frequently contain two types of errors: 1) missing content. a summary often leaves out important details such as medication dosage and route, undermining patients' medical self-management. 2) hallucination. summaries contain hallucinated content or content not supported by the input documents. for example, an abstractive summary on kidney infection was generated from an input document that describes urine infection. these types of errors are not uncommon in abstractive summarization @xcite @xcite , but they could be disastrous to patients. in this study, we build systems to facilitate detection and correction of those types of errors, allowing physicians to correct or edit system generated summaries. as illustrated in figure 1 , summarization produces a system summary; error alerting automatically detects errors from the generated after-visit summary. crucially, we build effective detectors with self-supervision on unlabeled data for error alerting. a novel dataset is constructed by synthesizing summaries containing medical events that are inconsistent with their source documents. ## related work recently there has been a lot of work on automatic summarization in the clinical domain: @xcite propose to generate the impression section of a radiology report using seq2seq models. @xcite perform image-to-text radiology report generation by optimizing entity-based rewards with reinforcement learning. studies are also performed for summarizing doctor-patient dialogues @xcite and evaluating system generated notes @xcite . early work has explored generation of hospital visit summaries using non-neural methods @xcite @xcite . recently, @xcite present the task of hospital-course summarization with the goal of generating a text to synthesize the hospital course. a crucial difference between our work and that of @xcite is we investigate a deep learning solution, whose primary focus is exposing neural abstractive summarizers to clinical notes and explicitly highlighting regions of a summary which need attention. this is in principle similar to checklist in @xcite . it is important for an after-visit summary generated from ehr notes to avoid type i and type ii errors. a type i error (false positive) suggests that there is false or inaccurate information in the summary, due to hallucinations, incorrect grounding, etc. it is a challenging and lingering problem facing natural language generation @xcite @xcite @xcite , despite remarkable recent progress @xcite @xcite @xcite . a more surprising observation is that the type ii error (false negative) is deemed particularly harmful to patients. when salient medical events such as diagnoses or treatments are left out of the after-visit summaries, it could have a detrimental effect on patients' self-care after being discharged from hospitals @xcite . this empirically motivates our work, where we seek to effectively identify salient medical events in ehr notes and alert physicians of any missing events to help them avoid those errors. a distinguishing characteristic of after-visit summaries is that they are patient-oriented. the summaries provide relevant and actionable information to patients, such as reasons for visit, diagnoses and procedures, etc. differing from physician-oriented ## summarization our method generates an after-visit summary from ehr notes concerning a patient. it is modelled as a single-document summarization task as the ehr notes were collapsed into a single document by the hospital and we were unable to recover individual ehr notes. we use @xmath0, ..., w n s } to denote tokens of the source document and @xmath1, ..., w n t } tokens of the target summary, n s and n t are length of the sequences. we explore a variety of summarization models to generate after-visit summaries. they are detailed in clinical notes are complex and full of references to medical events. however, the summary given to the patient is simple and clear. we are thus curious to know how medical events manifest themselves in the context of summarization. events are especially important for this task, as salient events happening at each medical encounter must be included in the after-visit summary. we define event nugget as a word or multi-word phrase that clearly expresses the occurrence of a medical event. event nuggets are identified by @xcite , an open-source software tool designed to discover medical concepts referred to in a text. each occurrence of the concept is assigned a concept unique identifier (cui) and its associated words are tagged in the text. in figure 3 , we show an example of medical concepts identified by metamap. further, those medical concepts are categorized into various semantic types. we focus on concepts pertaining to a selected set of entity and event types ( ## error detection and alerting event nuggets are associated with type i and type ii errors frequently found in the summary. a type i error indicates a summary contains a hallucinated fact or event that is not present in the source document. a type ii error suggests that an important medical event has been mistakenly left out of the summary, hampering its usability. in this section, we describe novel methods to detect likely errors and flag them in the text to alert clinicians. ## experiments in this section, we describe our dataset, perform indepth analyses on our models, and discuss feedback from physicians who participated in our qualitative evaluation. foot_1 ## discussion we discuss our findings from interviewing physicians and underline some of the key areas that are indispensable for further progress on this task. it would be unwise to truncate clinical notes, despite that most neural models use a fixed maximum length. e.g., a patient who underwent a heart transplant has a high risk of multiple medical comorbidities. it can lead to a large volume of ehr notes. interestingly, physicians tend to include more content in after-visit summaries if they believe patients have high medical literacy and are able to understand and act upon complex instructions. this indicates that future systems may produce summaries of varying length per patients' needs. a system generated summary: you were admitted for dizziness. you had a ct scan of your head which showed some thickening in the sinuses of your sinuses. you were seen by the ear nose and throat doctor who recommended that you take an antibiotic called unasyn while you are in the hospital. you also had an mri of your brain which did not show any stroke. you are doing better and can go home today. ## conclusion we tackle the problem of generation of patient aftervisit summaries. we compared state-of-the-art summarization models for this task and introduced a novel alerting mechanism to predict two types of errors, including missing medical events and hallucinations in summaries. extensive experiments using automatic metrics and expert evaluation show the effectiveness of our proposed approach.