Overview of the MEDIQA -Chat 2023 Shared Tasks on the Summarization & Generation of Doctor-Patient Conversations (2023)

Abstract: Automatic generation of clinical notes from doctor-patient conversations can play a key role in reducing daily doctors’ workload and improving their interactions with the patients. MEDIQA-Chat 2023 aims to advance and promote research on effective solutions through shared tasks on the automatic summarization of doctor-patient conversations and on the generation of synthetic dialogues from clinical notes for data augmentation. Seventeen teams participated in the challenge and experimented with a broad range of approaches and models. In this paper, we describe the three MEDIQA-Chat 2023 tasks, the datasets, and the participants’ results and methods. We hope that these shared tasks will lead to additional research efforts and insights on the automatic generation and evaluation of clinical notes.

## introduction recent progress in text summarization and generative ai can greatly benefit the healthcare system by automatically generating clinical notes from doctor-patient conversations. this can contribute to effective clinical care by reducing the doctors' workload to editing and validating the generated summaries/notes instead of writing the full notes during the consultations at the expense of their time or focus when talking and interacting with the patients. clinical note generation has seen an increased research interest in the recent years. for instance, @xcite tackled automatic medical scribing with dialogue2note sentence alignment and snippet summarization. @xcite introduced medicalsum, a guided clinical abstractive summarization model for generating medical reports from doctor-patient conversations. @xcite showed that in-domain pre-training improves clinical note generation from doctor-patient conversations. @xcite presented three user studies, on medical note generation systems and analyzed the clinicians' views of how the system could be adapted and improved. other efforts focused on the evaluation of medical note generation manually through consultation checklists @xcite or automatically using evaluation metrics that correlate with human judgments @xcite @xcite . @xcite introduced the primock57 collection of 57 mocked primary care consultations, one of the rare datasets dedicated to this task. the previous editions of the mediqa shared tasks focused on medical nlp tasks such as textual inference and question answering @xcite as well as the summarization of patient questions/answers and radiology reports @xcite . this third edition, mediqa-chat 2023 foot_0 , addresses the generation of clinical notes based on the summarization of doctor-patient conversations. all of the datasets and code created for this challenge are publicly available 2 . in this paper, we present the tasks and datasets in section 2 and section 3. in section 4, we present the evaluation methods and metrics used for the shared tasks. section 5 describes and discusses the participating teams' approaches and draws insights from the official challenge results. ## datasets the mts-dialog dataset, used in task a, consists of 1.7k pairs of conversations and associated summaries. the aci-bench dataset, used tasks b & c consists of 207 pairs of full doctor-patient conversations and associated clinical notes. ## evaluation in this challenge, we evaluated both the submitted runs and the submitted codes as described below. ## conclusion with the recent progress in large language models (llms), the mediqa-chat 2023 shared tasks provided an opportunity to evaluate the recently released llms (e.g., gpt-4, chatgpt) vs. older models (e.g., t5, bart) in order to develop sota models and approaches for the summarization and generation of doctor-patient conversations. the variety of runs submitted by the participating teams and the explored augmentation, fine-tuning, and prompting methods provided new insights on the best approaches and techniques for future research directions in domain-specific text generation. the best results in the summarization of short dialogues were obtained using a flan-t5 model that jointly predicts the section header and generates the section text (wanglab team). the team's approach on long dialogues also yielded the best challenge results using gpt-4 with in-context examples se-lected from task b training data. in task c, the best results were from the cadence team which leveraged a bart-large model fine-tuned on different datasets to generate conversations from clinical notes to augment tasks a and b training data. the newly introduced benchmarks allowed the organization of these shared tasks and the evaluation of the participating systems on unseen test sets. automatic evaluation remains an important and challenging task. in this edition, we relied on an ensemble of evaluation metrics and we added a new requirement to submit the code for a second evaluation of the outputs. we hope that these shared tasks will encourage further efforts towards automatic clinical note generation using recent ai advances to reduce the workload for medical professionals and to improve the quality and outcomes of doctor-patient encounters.