UMASS _ B io NLP at MEDIQA -Chat 2023: Can LLM s generate high-quality synthetic note-oriented doctor-patient conversations? (2023)

Abstract: This paper presents UMASS_BioNLP team participation in the MEDIQA-Chat 2023 shared task for Task-A and Task-C. We focus especially on Task-C and propose a novel LLMs cooperation system named a doctor-patient loop to generate high-quality conversation data sets. The experiment results demonstrate that our approaches yield reasonable performance as evaluated by automatic metrics such as ROUGE, medical concept recall, BLEU, and Self-BLEU. Furthermore, we conducted a comparative analysis between our proposed method and ChatGPT and GPT-4. This analysis also investigates the potential of utilizing cooperation LLMs to generate high-quality datasets.

## introduction the issue of the growing burden of clinical documentation has become a critical concern in healthcare, resulting in increased job dissatisfaction and burnout rates among clinicians and adversely affecting patient experiences. nevertheless, timely and accurate documentation of patient encounters is crucial for safe, effective care and communication between specialists. consequently, there is a growing interest in automating assisting doctors in diagnosis based on large language models (llms) due to its remarkable advancement in the field of artificial intelligence (ai), being highly sophisticated systems that have been extensively trained on massive amounts of textual data. @xcite @xcite @xcite the swift progress of ai and its extensive influence on various fields have garnered considerable attention from the research community. one notable area is the creation of instruction-following llms @xcite @xcite , which demonstrate extraordinary ability in understanding instructions and producing human-like responses. these auto-regressive llms undergo a two-step process: they are initially pre-trained on web-scale natural languages through next-token prediction and subsequently fine-tuned to comply with extensive human instructions @xcite . this method leads to impressive performances across a broad range of natural language processing (nlp) tasks and generalizes to unseen tasks, underscoring their potential as a comprehensive solution for diverse challenges, including natural language understanding, text generation, and conversational ai @xcite . many auto-regressive llms, such as chatgpt, have further training with rlhf to align with human preference and finally allow these models to generate content that most people prefer. in the biomedical domain, many researchers have attempted to apply auto-regressive models to medical tasks such as patient triage @xcite , automatic disease coding @xcite , and doctor-chatbot @xcite . however high-quality dialogue datasets featuring doctor-patient interactions are a task that is inherently complex. one major difficulty in constructing such a dataset is the sensitive nature of the content, as healthcare conversations often involve private and confidential patient information @xcite @xcite . ensuring privacy protection and adhering to strict data regulations, such as hipaa, becomes crucial in the development process. consequently, the compilation of authentic doctor-patient dialogues requires careful consideration of privacy and data protection measures to prevent potential ethical and legal concerns. recent work @xcite attempts to synthesize data by letting humans play the roles of doctor and patient in a conversation, but the huge cost makes the research community seek the help of llms or chatbot models to simulate such role-playing game (doctor and patient) for data augmentation. however, recent chatbot models @xcite are only based on singleturn or multi-turn question-and-answer repositories rather than real conversations between patients and doctors. question-and-answer datasets lack logical coherence, whereas real conversations can help the model understand the proper order of questions @xcite , thereby guiding patients to describe their symptoms accordingly and ultimately aiding in disease diagnosis. so we cannot simply use the recent medical chatbot models to generate high-quality note-oriented doctor-patient conversations. on the other hand, the success of these llm models remains heavily reliant on human input to guide conversations in the right direction. this dependence necessitates users to provide relevant and precise prompts based on their intentions and the chat agent's feedback, which can be challenging, time-consuming, and occasionally unfeasible. in healthcare contexts, individuals without medical expertise may struggle to generate appropriate prompts for directing communicative agents to deliver accurate medical advice or diagnoses @xcite . all of this raises a crucial question: how can we try role-playing games to guide conversations toward clinical note completion in healthcare settings without a large number of human annotations? to address these issues, we propose a cutting-edge cooperative agent framework, doctor-patient loop. this approach involves two main chatgpt agents taking on the roles of doctor and patient in dialogue, with additional chatgpt agents responsible for fact-checking, ensuring conversations remain focused on the provided notes, determining when the dialogue should be terminated, and refining the conversation to enhance its coherence and fluency. the collaboration among multiple chatgpt agents leads to the creation of more realistic doctor-patient dialogue datasets, which in turn can be utilized for training models that better mimic genuine healthcare communication scenarios. in this paper, we conducted a series of experiments with the help of the data set of the mediqa-chat competition shared task. specially, we present our entry for task-a and task-c. we explored a new approach to solve for task-a. we trained biomedlm foot_0 on the dataset of task-a and designed prompts for different section headers. for task-c, we explored the potential for creating scalable methods that promote autonomous cooperation among communicative agents in medical settings. we construct a doctor-patient loop to generate high-quality clinical dialogue. our paper's contributions can be summarized as follows: ## related work the mediqa-chat 2023 tasks 3 (abacha et al., 2023a; ben abacha et al., 2023; ben abacha et al., 2023; yim et al., 2023) focused on both di-alogue2note summarization and note2dialogue generation tasks. the researchers constructed a novel dataset comprising 1,700 doctor-patient conversations (16k turns and 18k sentences) and their summarized clinical notes (6k sentences). they also proposed an investigation of standard evaluation metrics, domain-specific metrics, and expert judgments for the task, including the calculation of the correlation between the automatic and manual scores for the evaluation of the generated clinical notes. in this paper, we use a cooperative agent framework to generate the conversation data sets. for the section header and content classification (task-a), the soap (subjective, objective, assessment, and plan) structure is commonly used by providers @xcite . the subjective section is a detailed report of the patient's current conditions, such as source, onset, and duration of symptoms, mainly based on the patient's selfreport. this section usually includes a history of present illness and symptoms, current medications, and allergies. the objective section documents the results of physical exam findings, laboratory data, vital signs, and descriptions of imaging results. the assessment section typically contains medical diagnoses and reasons that lead to medical diagnoses. the assessment is typically based on the content from the subjective and objective sections. the plan section addresses treatment plans based on the assessment. previous work focused on identifying these four general soap sections @xcite . in this paper, we focused on predicting the specific subsections. for the dialogue2note summarization task (task-a&b), there are many solutions already in the industry @xcite @xcite @xcite . this process generally follows a similar pipeline. clinical conversations are initially recorded and then transcribed. subsequently, the utterances are classified into various medical sections, and clusters of utterances containing medically relevant information for each section are predicted from the transcript. finally, a section-conditioned summarization model is employed to generate concise summaries for each cluster of utterances associated with their respective sections. however, the size of their private training data is larger than the training data of this competition. thus, we applied llm for this competition. instruction finetuning helps llm generalize into unseen tasks where training data is limited @xcite . for example, an instructionfinetuned 11b-param model outperforms the 60bparam model without instruction-finetuning in the big-bench dataset @xcite . thus, we instructed finetuned biomedlm model for task-a dialogue2note summarization task. 2022 ) studied the effectiveness and accuracy of llms for data annotation, respectively, achieving promising outcomes even when compared with crowd-workers. @xcite utilized llms to create positive sample pairs for training downstream models. at the same time, @xcite focused on generating appropriate prompts with llms to improve performance further. lastly, @xcite mainly targeted few-shot retrieval tasks, combining llms with a limited number of samples to produce additional training data for retrieval models. in the biomedical field, @xcite investigated the potential of llms in clinical text mining and introduced a novel training paradigm to address suboptimal performance and privacy concerns. @xcite examined responsible and ethical artificial intelligence generated content (aigc) in medicine, analyzing differences between humanauthored and llm-generated medical texts and developing machine learning workflows for efficient detection and differentiation. in this paper, we explored our cooperative agent framework's performance in task-c. ## experiment in this section, we discuss our proposed methods' performance on mediqa-chat 2023 task-a and task-c. all the detailed experiment settings can be found on our github. ## conclusion in conclusion, our work proposes innovative approaches to promote autonomous cooperation among communicative agents in medical settings to generate the doctor-patient dialogue datasets as valuable resources for developing ai systems that can better understand and address context-sensitive inquiries in healthcare communication. to validate the quality of our generated datasets, we establish a hybrid evaluation system consists of rouge scores, concept recall, sarebleurt and self-bleurt. we use our evaluation system to compare our model with gpt4 and chatgpt. we demonstrate that our method contributes to advancing the development of ai systems in healthcare communication and paves the way for more effective doctor-patient interactions.