Medically Aware GPT -3 as a Data Generator for Medical Dialogue Summarization (2021)

Abstract: In medical dialogue summarization, summaries must be coherent and must capture all the medically relevant information in the dialogue. However, learning effective models for summarization require large amounts of labeled data which is especially hard to obtain. We present an algorithm to create synthetic training data with an explicit focus on capturing medically relevant information. We utilize GPT-3 as the backbone of our algorithm and scale 210 human labeled examples to yield results comparable to using 6400 human labeled examples (~30x) leveraging low-shot learning and an ensemble method. In detailed experiments, we show that this approach produces high quality training data that can further be combined with human labeled data to get summaries that are strongly preferable to those produced by models trained on human data alone both in terms of medical accuracy and coherency.

## introduction with increasing usage of telehealth platforms @xcite , large scale ecosystems of providers and patients have become apparent. this has exacerbated the need for comprehensive visit summaries of the medical dialogues by the attending practitioner in order to facilitate accurate handoffs to other care providers or as a means of recording the interaction. however, having providers write summaries after each encounter is not only time consuming but also costly, limiting the scalability of telehealth platforms @xcite in these settings, an automated summarizer that can assist the practitioners can be extremely valuable. however, an important challenge of end-toend medical dialogue summarization is the lack of large scale annotated datasets. annotation of medical dialogues is expensive and slow because they need to be curated by trained experts. this is further compounded by the fact that labeled data may not be publicly shared because of patient privacy concerns and hipaa regulations. recent approaches to summarization @xcite use transfer learning where a pre-trained model (e.g. through self supervision of learning a language model) is fine tuned with a labeled dataset. however, fine-tuning still requires hundreds to thousands of labeled examples to obtain reasonable performance. methods such as @xcite aim to partially overcome these issues through modeling strategies that directly learn important inductive biases from smaller amounts of data. in addition, @xcite also handled data sparsity by leveraging a key insight of sequential nature of information flow in a medical dialogue: global summary of the dialogue can be composed from local dialogue turns (snippets). this enables collecting training data for snippets as opposed to the full conversation -an insight, we use in our paper as well. recently, openai developed gpt-3, a neural language model that is capable of natural language generation and completion of tasks like classification, question-answering, and summarization @xcite . the focus of that work is to enable task-agnostic and zero-shot or low-shot performance as opposed to a pre-trained model that needs to be fine-tuned separately on every downstream task. in this paper, we investigate the following question: how can a low-shot learner such as gpt-3 be leveraged to scale training data for medical dialogue summarization models? in answering this question within the context of gpt-3 as a black box proprietary api foot_0 , we took into account multiple considerations: taking into account these considerations, this paper makes the following contributions (figure 1 for a quick overview): the rest of the paper is structured as follows: § 2 discusses related work, § 3 explores whether gpt-3 can be used directly for medical summarization, § 4 introduces our approach, § 5 and § 6 describe our datasets and metrics respectively while § 7 illustrates our experiments. we end the paper with § 8 discussing our conclusions and future work. figure 1 : overview of our proposed approach: we train models on a mix of gpt-3-ens synthesized and human labeled data to get performance better than models trained on either of the sources 2 related work summarization emergence of sequence to sequence models and attention mechanisms @xcite has led to rapid progress on extractive @xcite , abstractive @xcite and hybrid models @xcite for summarization. much of the recent work has shown these models to generate near-human coherent summaries while retaining reasonable factual correctness. dialogue summarization: while most neural summarization has focused on news corpora, recent work has tried to tackle unique challenges associated with summarizing dialogues. (goo and chen, 2018) proposes using dialogue history encoders based on the type of dialogue section to inform the generation. @xcite propose using key points as a means of categorizing sections of dialogue. medical dialogue summarization existing work @xcite @xcite @xcite in this space focuses on effective summarization by incorporating medical knowledge from a modeling perspective. our work also focuses on incorporating medical knowledge from a data labeling perspective. we show how we leverage pretrained language models and low-shot learning @xcite to collect labeled data for medical dialogue summarization. we also show how this data can improve performance over models that are trained solely on existing human labeled data. ignoring the privacy concerns and practitioner-inthe-loop considerations, we first explore whether gpt-3 (brown et al., 2020) is a good medical summarizer by itself. gpt-3 takes as input a priming context to perform the task on a previously unseen example. priming context refers to the text description of a task and a few demonstrations of the task being accomplished (in our case, that would be dialogue snippet summarization). based on these observations, we might prematurely conclude that gpt-3 can not be used for medical summarization task. however, our key observation in exploring gpt-3 is that it is sensitive to the priming context (also reported in @xcite ), as the model does not learn but just adheres to the examples given. as we show in 4, we exploit this variability in gpt-3 output via ensembling and infusion of medical knowledge so that it can be used as a part of an effective low-shot learning approach to medical summarization. ## infusing medical knowledge in gpt-3 for use as a data generator we are interested in a model that uses only a small amount of human labeled data to learn an effective medical dialogue summarizer. at the same time, we want such a model to be used in a practical practitioner-in-the-loop setting where medical correctness and patient privacy are of paramount importance. in order to achieve these goals, we propose a two-pronged approach 1. introduce gpt-3-ens where we infuse medical knowledge into gpt-3 and use it within an inner loop to make it effective at medical summarization. ## datasets we collected a random subset of medical conversation dialogues from our chat-based telemedicine platform. often medical conversation follows a linear ordering of medical history gathering (understanding patient symptoms) that enables creating the summary of the dialog by stitching together summaries of the snippets in chronological order @xcite . therefore, we split each dialogue into a series of local dialogue snippets using a simple heuristic: the turns between two subsequent questions by a physician corresponds to a snippet. the length of these snippets ranged anywhere from two turns (a physician question and patient response) to ten turns. we had medical doctors foot_3 summarize these snippets. the doctors were asked to summarize the sections as they would for a typical clinical note by including all of the relevant history taking information. if a local snippet did not contain any history taking information it was excluded from annotations. for example in the beginning or end of conversations there may be turns that are purely greetings and not part of the patient history taking process. further some snippets maybe purely educational in nature and are excluded as well. we eventually obtained a total of 6900 labeled snippet-summary pairs. human labeled dataset train/test split: from the 6900 labeled snippet-summary pairs (denoted as h 6900 ), we generated a randomly sampled test set @xmath0. the dataset h 6900 -t is used to generate the priming dataset for gpt-3 related models as well as the datasets we use to train our summarization models. gpt-3-ens dataset: let gcf k p be the dataset of size p generated using gpt-3-ens with k ensembling trials. to generate dataset gcf @xmath1, we require {h n } k @xmath2, and thus n × k labeled examples for priming. these n × k examples are randomly sampled from the universe of human labeled examples h 6900 -t . in our experiments, we sample without replacement so that no examples are reused across the k tries. to allow comparison between our experiments with different k values, we use the same seed for random sampling. ## evaluation metrics multiple studies have shown that automated metrics in nlp do not always correlate well to human judgments as they may not fully capture coherent sentence structure and semantics @xcite . since medical dialogue summarization would be used to assist health care, it is important for doctors to evaluate the quality of the output. ## experiments and results additional models considered: to evaluate the efficacy of gpt-3-ens as a source of labeled data generator, we considered models with distinct objective functions for abstractive and hybrid (abstractive/extractive) summarization. we used pegasus @xcite for abstractive summarization and dr. summarize which we denote as drsum @xcite for extractive summarization. for drsum , we also use their best performing variant (referred as 2m-pgen in @xcite ) which penalizes generator loss and favors extractive copying. implementation details: we used gpt-3 via the api released by openai foot_6 . maximum response length was set to 128 tokens, temperature to 0.6 and presence and frequency penalties both set to 0. for gpt-3-ens , we use @xmath3, unless otherwise specified. we observed that @xmath4. we therefore fix the size of our priming dataset to be 21 in all experiments which invoke gpt-3. hence we set l to be a random subset of 210 examples from h 6900 -t . we followed parameter settings for dr-sum from @xcite for pretraining on the cnn-dailymail dataset. we then fine-tuned on our summarization task dataset with a batch size of 16, @xmath5, @xmath6.0, for two epochs with a learning rate of 0.15 using adagrad optimizer. we used the pegasus implementation that is pretrained on cnn-dailymail foot_7 provided by @xcite . we fine-tuned it on our summarization task dataset with an effective batch size of 256, @xmath7, @xmath8. for both pegasus and drsum , we used a beam size of four for decoding. ## conclusion we introduced a medically-aware gpt-3 data labeler, gpt-3-ens , for the task of medical conversation summarization. at the heart of the approach is a medically aware ensembling criterion that ensembles multiple summaries for an input from a powerful low-shot learner such as gpt-3. we showed that this approach can generate quality snippet model trained on h6400 model trained on gcf @xmath9: have you ever been tested for any underlying health conditions such as diabetes, hypothyroidism or polycystic ovarian syndrome? pt: no pt: i have been told i have prediabetes has not been tested for any underlying health conditions. hasn't tested for any underlying health conditions such as diabetes, hypothyroidism or polycystic ovarian syndrome has not been tested for any underlying health conditions. has been told has prediabetes. dr: dr: do you have pus appearing discharge from the site? pt: yes. if the bubbles pop it leaks out a watery substance has pus appearing from the site. pus appearing from the site pus discharge from the site. if bubbles pop it leaks out a substance.