Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniques (2021)

Abstract: Following each patient visit, physicians draft long semi-structured clinical summaries called SOAP notes. While invaluable to clinicians and researchers, creating digital SOAP notes is burdensome, contributing to physician burnout. In this paper, we introduce the first complete pipelines to leverage deep summarization models to generate these notes based on transcripts of conversations between physicians and patients. After exploring a spectrum of methods across the extractive-abstractive spectrum, we propose Cluster2Sent, an algorithm that (i) extracts important utterances relevant to each summary section; (ii) clusters together related utterances; and then (iii) generates one summary sentence per cluster. Cluster2Sent outperforms its purely abstractive counterpart by 8 ROUGE-1 points, and produces significantly more factual and coherent sentences as assessed by expert human evaluators. For reproducibility, we demonstrate similar benefits on the publicly available AMI dataset. Our results speak to the benefits of structuring summaries into sections and annotating supporting evidence when constructing summarization corpora.

## introduction electronic health records (ehr) play a crucial role in patient care. however, populating them can take as much time as attending to patients @xcite and constitutes a major cause of physician burnout @xcite . in particular, doctors document patient encounters with soap notes, semi-structured written accounts containing four sections: (s)ubjective information reported by the patient; (o)bjective observations, e.g., lab results; (a)ssessments made by the doctor (typically, the diagnosis); and a (p)lan for future care, including diagnostic tests, medications, and treatments. sections can be subdivided into 15 subsections. in a parallel development, patients increasingly record their doctor's visits, either in lieu of taking notes or to share with a family member. a budding line of research has sought to leverage transcripts of these clinical conversations both to provide insights to patients and to extract structured data to be entered into ehrs @xcite @xcite . in this paper, we introduce the first end-to-end methods for generating whole soap notes based on clinical conversations. our work builds on a unique corpus, developed in collaboration with abridge ai, inc. foot_0 ), that consists of thousands of transcripts of recorded clinical conversations together with associated soap notes drafted by a work force trained in the official style of soap note documentation. on one hand, this task is much harder than traditional summarization benchmarks, in part, because soap notes are longer (320 words on average) than summaries in popular datasets like cnn/dailymail @xcite , newsroom @xcite , and sam-sum @xcite (55, 27, and 24 words on average). on the other hand, our dataset offers useful structure: (i) segmentation of each soap note into subsections; and (ii) a set of supporting utterances that provide evidence for each sentence in the soap note. exploiting this structure, our methods outperform appropriate baselines. our first methodological contribution is to propose a spectrum of methods, for decomposing summarizaton tasks into extractive and abstractive subtasks. starting from a straightforward sequence-tosequence model, our methods shift progressively more work from the abstractive to the extractive component: (i) conv2note: the extractive module does nothing, placing the full burden of summarization on an end-to-end abstractive module. (ii) noteworthy for review of systems noteworthy for medications conversation (2) cluster (3) generate dr: good. and is it helping? do you have chest pains anymore? pt: no. no chest pains . pt: although i do still have some cough. dr: i see. and do you get, like, mucous with it or is it dry? pt: umm no it's usually dry. no mucous. dr: so are you taking the monteluekast regularly? pt: yeah, one everyday like you said. figure 1 : workflow of our best performing approach involving extraction and clustering of noteworthy conversation utterances followed by abstractive summarization of each cluster (fictitious data) ext2note: the extractive module selects all utterances that are noteworthy (i.e., likely to be marked as supporting utterances for at least one soap note sentence), and the decoder is conditioned only on these utterances; (iii) ext2sec: the extractive module extracts per-subsection noteworthy utterances and the decoder generates each subsection, conditioned only on the corresponding utterances; our best performing model, cluster2sent (figure 1 ), demands the most of the extractive module, requiring that it both select and group each subsection's noteworthy utterances. interestingly, we observe that given oracle (per-subsection) noteworthy utterances, a simple proximity-based 2 our code and trained models for the ami dataset: in summary, we contribute the following: ## related work summarization is a well-studied problem in nlp @xcite . while early works focused on simply extracting important content from a document @xcite , later approaches attempted to paraphrase the content into new sentences (abstractive summarization) @xcite @xcite . following the development of neural sequence models @xcite , more research focuses on neural generation of abstractive summaries @xcite @xcite . while many papers summarize news articles, others summarize conversations, in business meetings @xcite , customer service @xcite , and tourist information center @xcite contexts. in the space of two-step extractive-abstractive summarization approaches, @xcite summarize scientific papers by first extracting sentences from it and then abstractively summarizing them. chen and bansal (2018) extract important sentences from the input and then paraphrase each of them to generate the abstractive summary. while they assume that each summary sentence is supported by exactly one source sentence, in our medical conversations, many summary sentences synthesize content spread across multiple dialogue turns (e.g., a series of questions and answers). past work on abstractive summarization of medical conversations has focused on summarizing patient-nurse conversations with goals including capturing symptoms of interest @xcite and past medical history @xcite . these tasks are respectively similar to generating the review of systems and past medical history subsections of a soap note. in contrast, we aim to generate a full-length soap note containing up to 15 subsections, and propose methods to address this challenge by extracting supporting context for smaller parts and generating them independently. ## dataset we use two different datasets in this work. the primary medical dataset, developed through a collaboration with abridge ai, consists of doctor-patient conversations with annotated soap notes. additionally, we evaluate our summarization methods on the ami dataset @xcite , comprised of business meeting transcripts and their summaries. ## methods we investigate the following four decompositions of the summarization problem into extractive and abstractive phases, ordered from abstraction-heavy to extraction-heavy: conv2note takes an endto-end approach, generating the entire soap note from the entire conversation in one shot. ext2note first predicts all of the noteworthy utterances in the conversation (without regard to the associated section) and then generates the entire soap note in one shot from only those utterances. ext2sec extracts noteworthy utterances, while also predicting the section(s) for which they are relevant, and then generates each soap section separately using only that section's predicted noteworthy utterances. cluster2sent attempts to group together the set of noteworthy utterances associated with each summary sentence. here, we cluster separately among each set of section-specific noteworthy utterances and then generate each section one sentence at a time, conditioning each on the associated cluster of utterances. each of these pipelines leaves open many choices for specific models to employ for each subtask. for the abstractive modules of conv2note and ext2note, we use a pointer-generator network. the abstractive modules of ext2sec and cluster2sent, which require conditioning on section are modeled using conditioned pointergenerator networks (described in section 5), and fine-tuned t5 models which condition on the section being generated by means of prepending it to the input. t5 models could not be used in the conv2note and ext2note settings because their high memory requirement for long inputs could not be accommodated even with 48gb of gpu memory. for noteworthy utterance extraction, we primarily use a hierarchical lstm model and a bert-lstm model as described in the next section. all models are configured to have a scalar output for binary classification in ext2note, whereas for ext2sec and cluster2sent, they have multilabel output separately predicting noteworthiness for each section. note that the same utterance can be noteworthy with respect to multiple sections. we use the same trained utterance extraction models for both ext2sec and cluster2sent. for the clustering module in cluster2sent, we propose a heuristic that groups together any two supporting utterances that are close, meaning they have less than or equal to τ utterances separating them, where τ is a hyperparameter. this process is iterated, with the clusters growing in size by merging with other singletons or clusters, until every pair of close utterances have the same cluster membership. the value of τ is tuned on the validation set. since each cluster necessarily produces one sentence in the soap note, having too many or too few clusters can make the soap note too long or too short, respectively. therefore, for any given value of the hyper-parameter τ and any given section, the prediction thresholds of the extractor are tuned on the validation set to produce approximately the same number of clusters over the entire validation set as present in the ground truth for that section. among ground truth clusters containing multiple noteworthy utterances, 82% are contiguous. in an experiment where the heuristic is used to cluster the oracle noteworthy utterances for each section, and summaries are subsequently generated via the abstractive modules from cluster2sent, rouge-1 and rouge-2 metrics deteriorate by less than 1 point as compared to oracle clusterings (appendix table a3 ), demonstrating our heuristic's effectiveness. ## model architectures pointer-generator network we use the pointer-generator network introduced by @xcite for conv2note and ext2note. the model is a bidirectional lstm-based encoderdecoder model with attention. it employs a pointer mechanism to copy tokens directly from the input in addition to generating them by predicting generation probabilities for the entire vocabulary. the model also computes the weights that govern copying versus generating at each decoding timestep. ## experiments we first establish two baselines. randomnote randomly and uniformly samples a soap note from the training set and outputs it as the summary for any input conversation. oracleext presents all the ground truth noteworthy utterances (evidence) from the conversation as the soap note without any abstractive summarization. thus, the oracleext baseline has the advantage of containing all the desired information (e.g., names of medicines) from the conversation, but the disadvantage of not being expressed in the linguistic style of a soap note which leads to lower n-gram overlap. the opposite is true for the randomnote baseline. both baselines give similar performance and are outperformed by the simple conv2note approach ( we train the abstractive modules for the 4 approaches described in section 4 with the ground truth noteworthy utterances as inputs. to estimate an upper bound on the performance we can reasonably hope to achieve by improving our noteworthy utterance extractors, we test our models with oracle noteworthy utterances in the test set. all algorithms relying on oracle noteworthy utterances outperform conv2note, and exhibit a monotonic and significant rise in rouge scores as we move towards the extraction-heavy end of the spectrum ( for predicting noteworthy utterances, we use two baselines: (i) logistic regression on tf-idf utterance representations; and (ii) a model with a bidirectional lstm to compute token-averaged utterance representations, followed by a linear classification layer. these two models make the predictions for each utterance independent of others. in contrast, we also use models which incorporate context from neighboring utterances: (a) a hierarchical lstm; and (b) a bert-lstm model as described in section 5. the latter two methods perform much better ( using predicted noteworthy utterances and clusters instead of oracle ones leads to a drop in rouge scores, but the performance of ext2sec and cluster2sent is still better than conv2note ( ## qualitative analysis the conditioned pointer-generator and t5 models used in cluster2sent learn to place information regarding different topics in appropriate sections. hence, given a cluster of supporting utterances, the models can generate different summaries for multiple sections (figure 2 ). for example, given the same supporting utterances discussing the patient's usage of lisinopril for low blood pressure, a model generates "low blood pressure" in the review of systems section, and "lisinopril" in medications section. we direct the reader to the appendix for examples of full-length generated soap notes. interestingly, when the abstractive model is given a cluster of utterances that are not relevant to the section being generated, the model sometimes outputs fabricated information relevant to that section such as saying the patient is a non-smoker in social history, or that the patient has taken a flu shot in immunizations . hence, the quality of produced summaries heavily depends on the ability of the extractive step to classify the extracted utterances to the correct section. another cause of false information is the usage of pronouns in clusters without a mention of the referred entity. in such situations, t5 models frequently replace the pronoun with some arbitrary entity (e.g. "she" with "daughter", compounds with "haemoglobin", and medicines with "lisinopril"). occasionally, the abstractive module produces new inferred information that is not mentioned explicitly in the conversation. in one instance, the model generated that the patient has a history of heart disease conditioned on a cluster that men-tioned he/she takes digoxin, a popular medicine for heart disease. similarly, the model can infer past medical history of "high cholesterol" upon seeing pravastatin usage. such inferences can also lead to incorrect summaries, e.g., when a doctor explained that a patient has leaky heart valves, a model added a sentence to the diagnostics and appointments section saying "check valves". of the conversation independently, which may lead to contradictions in the soap note. in one visit, the patient was asked about chest pain twice-once in the beginning to get to know his/her current state, and once as a question about how he/she felt just before experiencing a fall in the past. this led to the model generating both that the patient denied chest pain as well as confirmed chest pain, without clarifying that one statement was for the present and another for the past. ## human evaluation we asked trained human annotators to evaluate generated soap notes for 45 conversations. every sentence in each soap note was labeled according to various quality dimensions such whether it was factually correct, incoherent, irrelevant, redundant, or placed under an inappropriate section. the detailed statistics of annotations received for each quality dimension are provided in the appendix. we also collected aggregate annotations for the comprehensiveness of each soap note and the extent to which it verbatim copied the transcript on a 5-point likert scale. the summaries generated by three methods (conv2note(pointergenerator), cluster2sent(pointer-generator) and cluster2sent(t5-base)) were presented in random order to hide their identities. for each sentence, we asked for (i) factual correctness of the sentence; (ii) if the statement is simply repeating what has already been mentioned before; (iii) if the statement is clinically irrelevant; (iv) if the statement is incoherent (not understandable due to grammatical or semantic errors); and (v) if the statement's topic does not match the section in which it is placed. in addition, we asked two separate questions for rating the overall summary on a scale of 1-5 for its (i) comprehensiveness and (ii) extent of verbatim copying from conversation. the human evaluation of the soap notes was done by workers who had also participated in the creation of the dataset of soap notes. hence, they had already been extensively trained in the task of soap note creation, which gave them appropriate knowledge to judge the soap notes. to quantify the performance among different methods, we consider a scenario where each generated soap note has to be post-edited by discarding undesirable sentences. for a generated soap note, we define its yield as the fraction of its total sentences that are not discarded. the sentences that are retained are those that are both factually correct and were not labeled as either repetitive or incoherent. the human annotations show that both cluster2sent-based methods tested produced a higher yield than the conv2note baseline (@xmath0.02). t5-base performs better than conditioned pointer-generator as the abstractive module in cluster2sent setting, producing significantly more yield ( ## conclusion this paper represents the first attempt at generating full-length soap notes by summarizing transcripts of doctor-patient conversations. we proposed a spectrum of extractive-abstractive summarization methods that leverage: (i) section-structured form of the soap notes and (ii) linked conversation utterances associated with every soap note sentence. the proposed methods perform better than a fully abstractive approach and standard extractiveabstractive approaches that do not take advantage of these annotations. we demonstrate the wider applicability of proposed approaches by showing similar results on the public ami corpus which has similar annotations and structure. our work demonstrates the benefits of creating section-structured summaries (when feasible) and collecting evidence for each summary sentence when creating any new summarization dataset.