Exploring Robustness in Doctor-Patient Conversation Summarization: An Analysis of Out-of-Domain SOAP Notes (2024)

Abstract: Summarizing medical conversations poses unique challenges due to the specialized domain and the difficulty of collecting in-domain training data. In this study, we investigate the performance of state-of-the-art doctor-patient conversation generative summarization models on the out-of-domain data. We divide the summarization model of doctor-patient conversation into two configurations: (1) a general model, without specifying subjective (S), objective (O), and assessment (A) and plan (P) notes; (2) a SOAP-oriented model that generates a summary with SOAP sections. We analyzed the limitations and strengths of the fine-tuning language model-based methods and GPTs on both configurations. We also conducted a Linguistic Inquiry and Word Count analysis to compare the SOAP notes from different datasets. The results exhibit a strong correlation for reference notes across different datasets, indicating that format mismatch (i.e., discrepancies in word distribution) is not the main cause of performance decline on out-of-domain data. Lastly, a detailed analysis of SOAP notes is included to provide insights into missing information and hallucinations introduced by the models.

## introduction automatically generated summary notes of doctorpatient conversations could improve the healthcare system. first, the generated notes serve as a valuable resource, allowing doctors to review and validate the information from the conversation with a patient, ensuring that vital information is noticed. in addition, the summary notes can be integrated into hospitalization risk prediction models @xcite , empowering healthcare professionals with data-driven insights to make more precise clinical decisions. however, summarizing doctor-patient conversations poses distinct challenges owing to its specialized domain. specifically, medical conversations often involve highly specialized terminology that requires domain-specific knowledge to understand and summarize accurately. in addition, it is preferable to structure the generated note with subjective (information reported by the patient), objective (objective observations), assessment (doctor's evaluation), and plan (future care plan) (soap). soap format is preferable because it is widely utilized by healthcare providers to document a patient's progress, providing an organized framework that reduces communication confusion among healthcare professionals. these challenges hinder the direct application of general-purpose summarization techniques to doctor-patient conversations, underscoring the need for a specialized model. doctor-patient conversation summarization has attracted significant attention recently @xcite @xcite @xcite . in 2023, the mediqa-chat challenge @xcite attracted 120 registered teams from the academy and industry. although various methods are proposed in mediqa-chat, it remains a challenging field that needs further investigation. first, mediqa-chat focuses on in-domain training and testing. however, cross-dataset analysis for doctor-patient conversation summarization is crucial because collecting in-domain training data is usually challenging given the constraints imposed by privacy and security concerns. second, a detailed assessment of performance across soap note categories is essential. such insights into the performance of each category can play a pivotal role in developing improved model structures and designing more effective evaluation metrics. in this study, we investigate cross-dataset performance of state-of-the-art (sota) doctor-patient summarization models. our focus is on generative summarization models because the real-world clinical notes are in an abstractive format. the experiments were conducted on english datasets as the setting of most previous studies. the results of soap notes are evaluated separately to gain a deeper understanding of the strengths and limitations of the current models. we hope our result can offer new insights for future research in developing a robust doctor-patient summarization model for real-world scenarios. ## related work the mediqa-chat challenge (abacha et al., 2023a) separated doctor-patient conversation summarization into different tasks. models designed for task a predict the topic category of the conversation and then generate notes. the task a models are closer to a general-purpose summarization model, producing notes without specifying distinct sections. in the top performance models, wanglab @xcite fine-tuned a flan-t5 model @xcite for summarization and note classification. summqa @xcite used biobert @xcite to support the section classification, minilm @xcite to select the prompt for gpt4, and gpt4 to predict the section class and generated the final note. the cadence @xcite model fine-tuned bart-large on the samsum dataset, followed by fine-tuning on the augmented dataset. in addition, a n-pass summarization was employed to handle long conversations. models designed for task b are soap-oriented, generating notes with soap sections. in the top performance models, wanglab used instructor @xcite to select the top-k conversation that is similar to the testing data, then used the selected conversations and notes as the incontext learning examples for gpt4. they also achieved top performance with the fine-tuned longformer encoder-decoder (led) @xcite . summqa @xcite used the minilm @xcite to select the prompt for the gpt4 in-context learning examples as their model for task a. gersteinlab @xcite used gpt-4 with specifically designed instruction. task a in the mediqa-chat challenge was evaluated on the mts-dialog dataset @xcite , which has a relatively shorter conversation and reference notes related to a specific category. task b was focused on the aci-bench @xcite dataset, which has a relatively longer conversation and a long note with soap sections. most top-performance teams in task a used fine-tuning language model (lm)-based methods, while most top-performance teams in task b introduced gptbased approaches. the results seem to indicate that the fine-tuning llm-based method is more suitable for short dialogues with a specific category of information. in contrast, the gpt-based method is preferable for the long dialogue with detailed soap information @xcite . however, in real-world scenarios, conversations may vary in length and encompass one or multiple categories of information. therefore, in this study, we aim to understand how these models perform in an cross-dataset settings and identify potential errors made by the models. ## data we use two open-source doctor-patient conversation datasets, mts-dialog @xcite and aci-bench @xcite . both datasets contain doctor-patient conversations, the corresponding note of the conversation, and the category of the note. figure 1 illustrates the samples in the two datasets, and compared with the two datasets, the mts-dialog dataset contains relatively shorter conversation, and the reference note follows a concise format, comprising either a few words or a oneparagraph structure with a section header specifying the note category. in contrast, the conversations in the aci-bench dataset are relatively longer, and the reference notes includes all soap sections. we categorized the note in the mts-dialog dataset and divided the note in aci-bench dataset into s, o, or ap categories for analysis. note that we merged a and p as ap because these aci-bench mts-dialog dialogue doctor: hi, andrew. how are you? patient: hey, good to see you. doctor: i'm doing well, i'm doing well. … doctor: let me know if your symptoms worsen and we can talk more about it, okay? patient: you got it. doctor: all right. hey, dragon. finalize the note. doctor: good afternoon, sir. my chart here says that you are a fifty one year old white male, is that correct? patient: good afternoon, doctor. yes, all of that is correct. ... doctor: finally, your ecog score is one according to the nurse, is that correct? patient: yes, doctor. that's correct. note chief complaint upper respiratory infection. history of present illness andrew campbell is a 59-year-old male with a past medical history significant for depression, … section header: genhx section text: a 51-year-old white male diagnosed with ptld in latter half of 2007. he presented with symptoms of increasing adenopathy, abdominal pain, weight loss, and anorexia. …. are merged into ap in the aci-bench dataset, making it difficult to separate them into a and p. ## methods we divided the summarization model for doctorpatient conversation into general and soaporiented configurations (illustrated in figure 2 ). in this study, we investigate the current sota models of each configuration in a cross-dataset setting. our research question is: rq1: how do current sota doctor-patient conversation summarization models perform on out-of-domain datasets, and what causes the performance decline? ## limitations one limitation of this study is that the soap data in the mts-dialog dataset is unbalanced, with most references focusing on subjective information. in addition, real-world doctor-patient conversations are complex in size and medical specialties and cannot be fully represented by two datasets. another issue lies in the generative model producing varied results in different runs, and the performance of the gpt model is affected by the prompt. ## conclusion in this study, we evaluated the sota doctor-patient summarization models on out-of-domain data and investigated the challenges of using fine-tuning lm and gpt-based summarization models in realworld applications. for a model with a general configuration, the results indicate a high tendency of omitting objective information in the generated note. this concern can be alleviated by adopting the soap-oriented configuration, which orients the model to generate information relevant to all essential categories. despite achieving the highest performance on in-domain data, the fine-tuned lm with soap-oriented configuration exhibits a significant hallucination issue. to generate a note closer to its training data, the model produces hallucinations when none or insufficiently related information is present in the conversation. in contrast, limitations of gpt-based models arise from a tendency to offer their own suggestions for the assessment and plan. we hope our results provide insights for future work toward creating more robust models for real-world settings.