N ote C hat: A Dataset of Synthetic Patient-Physician Conversations Conditioned on Clinical Notes (2024)

Abstract: We introduce NoteChat, a novel cooperative multi-agent framework leveraging Large Language Models (LLMs) to generate patient-physician dialogues. NoteChat embodies the principle that an ensemble of role-specific LLMs, through structured role-play and strategic prompting, can perform their assigned roles more effectively. The synergy among these role-playing LLMs results in a cohesive and efficient dialogue generation. Evaluation on MTS-dialogue, a benchmark dataset for patient-physician dialogues-note pairs, shows that models trained with the augmented synthetic patient-physician dialogues by NoteChat outperforms other state-of-the-art models for generating clinical notes. Our comprehensive automatic and human evaluation demonstrates that NoteChat substantially surpasses state-of-the-art models like ChatGPT and GPT-4 up to 22.78% by domain experts in generating superior synthetic patient-physician dialogues based on clinical notes. NoteChat has the potential to engage patients directly and help clinical documentation, a leading cause of physician burnout.

## introduction clinical dialogue is an essential part of clinical workflow. clinical documentation is a two-step process. it first engages patients through conversation to collect patient-specific information such as demographic information, family history of diseases, and signs and symptoms and then generates electronic health records (ehrs) from the dialogues. currently clinical documentation is mainly done by physicians at both steps, a labor intensive process that contributes to physician burnout, defined as a state of emotional, physical, and mental ex1 our synthetic patient-physician dialogue data is in supplementary material and will be publicly available together with all codes and prompts upon acceptance. ## human evaluation to assess the quality of synthetic conversations generated by different methods (chatgpt, gpt-4, notechat), we conducted a human evaluation using crowd-sourcing and domain experts. ## https://sites.google.com/view/mediqa2023 to the best of our knowledge, we are the first to introduce a large-scale publicly available patientphysician conversation dataset in english, each accompanied by corresponding medical documents, with an average number of utterances exceeding 20 rounds. in addition, our extrinsic eval shows that the notechat can be used as auxiliary data for conversation2note or note2conversation tasks and can also be used as a synthetic medical dialogue dataset alone to engage patients directly and help clinical documentation @xcite @xcite @xcite . multiple llms cooperation: our work builds upon the recent advances in deploying two llms as cooperative agents (panait and luke, 2005) for multi-round conversation generation. in particular, notechat is inspired by camel (li et al., 2023a), which assigns roles to two llms (e.g. student and teacher) in order to facilitate conversation between the two agents for a particular task (e.g. teaching). similar to camel's findings, we found that roleplay by itself may hallucinate or generate fake replies that repeat most of the previous utterances. to solve this issue, we proposed a novel planning module to ground agents to certain keywords. cho et al. (2023) also addresses the challenges of using llm to craft a dialogue dataset with specified personas. they emphasize the importance of grounding and context in conversation generation. similarly, notechat relies on structured clinical notes segmented using the soap format to provide context for our dialogue synthesis to diagnose a patient. however, their work is limited to generating open-domain dialogue, while we focus on task-oriented dialogue. 6 conclusion in this study, we present notechat, a cooperative multi-agent framework leveraging llms for generating synthetic patient-physician conversations conditioned on clinical notes. notechat consists of planning, roleplay, and polish modules. extensive evaluations demonstrate that notechat facilitates high-quality synthetic patient-physician conversations, underscoring the untapped potential of llms in healthcare and offering promising avenues for the intersection of ai and healthcare. 8 formed a preliminary study to generate synthetic conversation from case reports within research articles indexed from january 2002 to july 2022 by pubmed central. the credibility of these case reports is ensured as they are peer-reviewed and published in academic journals. moreover, the type of disease is diverse as they are sourced from various hospital departments and are not limited to intensive care units (such as mimic). thus, models trained using our synthetic data may benefit from these characteristics. ## 15196 in this task, we ask for your expertise in annotating the quality of the system-generated dialogues by machine learning models. mainly we provide the ground truth dialogue and the clinical note along with system-generated dialogues and ask for your preference. output your ranking for system-generated dialogues. use the following format, and do not add any other text. some examples: @xmath0: [clinical note] ground truth dialogue: [dialogue] system-generated summaries: 1. [dialogue1] 2. [dialogue2] 3. [dialogue3] now, output your ranking: