S umm QA at MEDIQA -Chat 2023: In-Context Learning with GPT -4 for Medical Summarization (2023)

Abstract: Medical dialogue summarization is challenging due to the unstructured nature of medical conversations, the use of medical terminologyin gold summaries, and the need to identify key information across multiple symptom sets. We present a novel system for the Dialogue2Note Medical Summarization tasks in the MEDIQA 2023 Shared Task. Our approach for sectionwise summarization (Task A) is a two-stage process of selecting semantically similar dialogues and using the top-k similar dialogues as in-context examples for GPT-4. For full-note summarization (Task B), we use a similar solution with k=1. We achieved 3rd place in Task A (2nd among all teams), 4th place in Task B Division Wise Summarization (2nd among all teams), 15th place in Task A Section Header Classification (9th among all teams), and 8th place among all teams in Task B. Our results highlight the effectiveness of few-shot prompting for this task, though we also identify several weaknesses of prompting-based approaches. We compare GPT-4 performance with several finetuned baselines. We find that GPT-4 summaries are more abstractive and shorter. We make our code publicly available.

## introduction medical dialogue summarization is a long-standing challenge in nlp @xcite @xcite . medical scribes write notes on doctor-patient conversations in a predefined template called soap notes @xcite , which contains sections for information from the patient, test results and observations, diagnosis, and the conclusion or treatment. medical summarization is challenging for several reasons. it requires dialogue understanding, where data is often limited @xcite ; this is compounded by the sensitive nature of medical information, which restricts the release of training data for this task @xcite . doctors and patients may discuss several conditions in the same conversation, requiring the scribe to differentiate @xcite . scribes often use medical terminology in the notes that are not present in the doctorpatient conversation @xcite . additionally, medical summarization is a high-stakes domain @xcite , motivating several efforts to build explainabile systems for this task @xcite . in parallel, research on large language models (llms) has demonstrated compelling few-shot capabilities across domains @xcite . in this paper, we explore several potential applications of a recent llm, @xcite , on medical summarization. we use gpt-4 and finetuned biobert @xcite as an ensemble for classifying the section headers of medical summaries, a 20-category classification problem. then, given a candidate section header, we apply maximal marginal relevance (mmr) @xcite to select examples for a fewshot demonstration and use these examples to prompt gpt-4 for section-wise summarization. this approach outperforms finetuning bart @xcite and t5 @xcite over the limited available data. for full-note summarization, we take a similar approach, but select only a single example for the demonstration due to the increased length of the inputs. this also outperforms our supervised baselines. we outline several additional potential prompting approaches and compare their relative efficacy. applying llms for medical summarization is a compelling solution to the data scarcity problems in this domain, and we find promising performance, with our team placing second in the mediqa 2023 shared task for subtask a and division summary for subtask b. however, we also identify key areas for improvement. we analyze the differences in outputs between the settings in output length and extractive ability. we find that the summaries generated by llms tend to be shorter and less extractive than human-generated summaries as well as sota fine-tuned biomedical summarization models. we also note the impracticality of this approach for real data, due to privacy concerns. ## background dialogue2note summarization was one of two tracks in the mediqa-chat 2023 shared task @xcite . the track was further comprised of two tasks. task a involves generating a section-specific clinical summary from a conversation between a patient and a doctor. additionally, task a includes a classification task: assigning each dialogue an appropriate section header. there are 1200 conversations in the training split of the dataset (ben abacha et al., 2023) for task a. the distribution over the section headers in task a is a long-tailed distribution, displayed in figure 2 . task b involves generating a full note summary given a conversation; these summaries were evaluated on the section-level and the full-note level. there were 67 conversations in the training split of the dataset @xcite ; these dialogues and reference summaries are significantly longer than those for task a, as these dialogues encompass an entire conversation between a patient and a doctor. the distribution of the difference in dialogue and summary length for both tasks is shown in figure 1 . maximal marginal relevance was created to reduce redundancies in multi-document summaries @xcite . @xcite used mmr to generate query-focused summaries from pre-trained models without performing fine-tuning. @xcite use mmr to select examples for in-context prompting. the success of prompt-based models such as gpt-3 (brown et al., 2020) has allowed for learning from natural language task instructions and/or a few demonstrative examples in the context without updating model parameters. in news summarization, @xcite find that gpt-3 summaries were preferred by humans over summaries from fine-tuned models trained on large summarization datasets; they posit that zero-shot summaries avoid pitfalls from low-quality training data that are common in summaries from fine-tuned models. in the biomedical domain, pre-trained language models and few-shot learning has been used to collect and generate labeled data for medical dialogue summarization @xcite . recent work has used gpt-4 to pass the usmle without any specialized prompt crafting @xcite and perform zero-shot medical evidence summarization across six clinical domains @xcite . few-shot learning few-shot learning can be unstable as the prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-ofthe-art @xcite . recent work on prompting has tried to mitigate these problems through techniques such as calibration @xcite , prompt combination @xcite , or auto-matic prompt generation @xcite . to mitigate any instability caused by a model's bias, @xcite estimated the bias towards each answer by asking for its prediction when given the training prompt and a content-free test input such as "n/a" and then fit calibration parameters that cause the prediction for this input to be uniform across answers. to date, studies in prompt combination are rooted in paraphrasing-based methods that take a seed prompt and paraphrase it into several semantically similar expressions. typically simple ensemble methods @xcite such as maximal marginal relevance @xcite are used to combine the answers to the different prompts as to provide each prompt to contribute to the final answer. a number of techniques have also been proposed for selecting fewshot examples @xcite . fewshot techniques often rely on selecting optimal examples from a large dataset; some work has shown that this leads to an overstatement of fewshot performance, as a large number of labeled examples are necessary to select good examples for the fewshot prompt @xcite . we note that we use the full datasets (1,200 examples for task a, 40 for task b) for our prompt selection techniques. ## methodology and baselines our summary generation pipeline remains the same across the two tasks: we use gpt-4 to generate a summary given k in-context examples. ## future work and conclusion this paper attempts to automatically generate summaries or structured soap notes from a conversation between a doctor and a patient. we tackle this problem by generating section-wise summaries, classifying these summaries into appropriate section headers and generating full-length summaries from longer conversations. we conclude from the results that promptingbased techniques by themselves can not perform optimally themselves but do give some outstanding results when combined with existing techniques, like prompt selection using minilm. next, we also dive deep into where prompting-based methods underperform the standard models like bart and t5. overall, our model concluded 3rd amongst all runs submitted and 2nd as a team for task a, which aimed at producing section-level summaries. our system also stood 4th amongst all runs submitted and 2nd as a team in division-wise summaries for task b. in the future, we plan to use an ensemble of extractive and abstractive methods of generating summaries as well as using diversity algorithms