Generating medically-accurate summaries of patient-provider dialogue: A multi-stage approach using large language models (2023)

Abstract: A medical provider’s summary of a patient visit serves several critical purposes, including clinical decision-making, facilitating hand-offs between providers, and as a reference for the patient. An effective summary is required to be coherent and accurately capture all the medically relevant information in the dialogue, despite the complexity of patient-generated language. Even minor inaccuracies in visit summaries (for example, summarizing “patient does not have a fever” when a fever is present) can be detrimental to the outcome of care for the patient. This paper tackles the problem of medical conversation summarization by discretizing the task into several smaller dialogue-understanding tasks that are sequentially built upon. First, we identify medical entities and their affirmations within the conversation to serve as building blocks. We study dynamically constructing few-shot prompts for tasks by conditioning on relevant patient information and use GPT-3 as the backbone for our experiments. We also develop GPT-derived summarization metrics to measure performance against reference summaries quantitatively. Both our human evaluation study and metrics for medical correctness show that summaries generated using this approach are clinically accurate and outperform the baseline approach of summarizing the dialog in a zero-shot, single-prompt setting.

## introduction a critical clinical task during a medical encounter between a patient and a physician is summarizing the conversation. this summarized note, whether created by a physician or medical assistant, contains important information about the visit and serves as a reference for future patient visits and for the patient. physicians often spend many hours each week performing such tasks. charting work, in general, has been identified as a contributing factor to increased rates of physician burnout (eschenroeder et al., 2021). automating medical conversation summarization has been studied with limited success (pivovarov and elhadad, 2015; liang et al., 2019; gao et al., 2022; macavaney et al., 2019; chintagunta et al., 2021). some methods try to directly summarize the chat (enarvi et al., 2020 @xcite . as base deep learning methods have improved and pre-trained language models specific to summarization such as pegasus (zhang et al., 2020), bart @xcite , and gpt-3 @xcite have emerged, we have seen increased fidelity of the summaries generated. however, performance is still not to a reliable standard in prac-tical settings for several reasons. first, the lack of labeled clinical data makes it hard to build highperformance fine-tuned models. this reflects lowerthan-expected specificity and accuracy in faithfully capturing medical concepts and their affirmations (e.g., present, absent, unknown). second, customtrained models need more world knowledge to understand patient language in these conversations and how they map to medical concepts. third, these models often require breaking conversations into smaller segments to deal with limited context windows. this in turn introduces challenges such as incorrect anaphora and coreference resolution across segmented pieces of the conversation. in both qualitative physician analysis of medical dialogue summaries and quantitative metrics, medsum-ent generates clinically accurate summaries and produces summaries that are preferable to a zero-shot, single prompt baseline. ## methods we now detail the components of our medsum-ent framework for medical dialogue summarization, represented in figure 1 . ## results in our expert human evaluations, figure 2 demonstrates medsum-ent (5-shot, semantic) summaries are preferred over the baseline summaries 66% to 34%. our expert evaluators also rate medsum-ent capturing all relevant medical information in 40% of evaluated summaries, most information in 48%, some information in 12%, and zero information in 0%. this provides further qualitative evidence for medsum-ent's ability to effectively summarize. however, our expert evaluators also rate 28% of the summaries evaluated as containing incorrect information that could harm the patient if acted on by medical providers. often these are due to misattributed symptoms and conditions (e.g., symptoms marked as absent but were present, missed medication allergies). this is consistent with the gpt-f1 measures for pertinent positives and negatives in ## conclusion we introduce medsum-ent, a multi-stage framework for medical dialogue summarization that modularizes summarization into multiple stages that extract and refine medical entities from dialogue turns. through human evaluation and quantitative metrics, we show that this method is clinically accurate and preferable to naive zero-shot summarization with gpt-3. we hope that future work can investigate refinement modules and iterative summarization further and conduct wider expert human evaluation studies to better understand challenges in bringing model-assisted summarization to medical providers in the near term.