An Empirical Study of Clinical Note Generation from Doctor-Patient Encounters (2023)

Abstract: Medical doctors spend on average 52 to 102 minutes per day writing clinical notes from their patient encounters (Hripcsak et al., 2011). Reducing this workload calls for relevant and efficient summarization methods. In this paper, we introduce new resources and empirical investigations for the automatic summarization of doctor-patient conversations in a clinical setting. In particular, we introduce the MTS-Dialog dataset; a new collection of 1,700 doctor-patient dialogues and corresponding clinical notes. We use this new dataset to investigate the feasibility of this task and the relevance of existing language models, data augmentation, and guided summarization techniques. We compare standard evaluation metrics based on n-gram matching, contextual embeddings, and Fact Extraction to assess the accuracy and the factual consistency of the generated summaries. To ground these results, we perform an expert-based evaluation using relevant natural language generation criteria and task-specific criteria such as critical omissions, and study the correlation between the automatic metrics and expert judgments. To the best of our knowledge, this study is the first attempt to introduce an open dataset of doctor-patient conversations and clinical notes, with detailed automated and manual evaluations of clinical note generation.

## introduction the recent progress in automatic summarization has been highly influenced by large transformerbased language models and the availability of largescale datasets. the summarization of medical conversations is well positioned to benefit from similar approaches, but is facing domain-and task-specific obstacles such as the lack of data and relevant evaluation protocols. medical doctors spend on average 52 to 102 minutes per day writing clinical notes from their conversations with the patients @xcite . the time spent with electronic health record systems contributes to work-life imbalance, dissatisfaction, high rates of attrition, and a burnout rate exceeding 50% @xcite . summarization models could play a key role in reducing that workload by generating clinical notes from the doctorpatient encounters @xcite . summarizing doctor-patient conversations for a clinical setting brings its own set of challenges and nuances in addition to the typical natural language understanding and generation components. for instance, omission of critical medical facts is likely to alter patient outcomes and should be one of the critical/deciding factors in adopting one summarization system over another. hallucinations are also likely to impact the clinical outcome if they are not avoided (or detected with a high accuracy). designing and improving summarization models that address these challenges can benefit from wider research efforts on the task. however, the lack of publicly available doctor-patient dialogue datasets limits wider research efforts from the nlp community in this summarization task. in this paper, we tackle the lack of data for the task by building a new dataset of doctor-patient conversations and associated clinical notes. we avoid privacy infringement risks, by creating simulated conversations from publicly available clinical notes. our main contributions are: (i) a new dataset of 1,700 doctor-patient conversations (16k turns and 18k sentences) and their summarized clinical notes (6k sentences). to the best of our knowledge, this is the first publicly-available dataset of medical conversations and associated notes at this scale, (ii) an evaluation of several sota summarization models, including variants that use existing relevant datasets, augmented training data, and guided summarization for medical conversation summarization, and (iii) a study of standard evaluation metrics, domain-specific metrics, and expert judgments for the task, including computing the corre-lation between the automatic and manual scores for the evaluation of the generated clinical notes foot_0 . dialogue summarization is relatively less studied in open domain, with a few efforts tackling the summarization of conversations and meetings @xcite @xcite and the creation of meeting summarization datasets @xcite . medical conversation summarization is also under-studied, except for a few recent efforts @xcite @xcite . for instance, @xcite applied a pointer generator model to generate summaries from doctorpatient dialogues in telemedicine. instead of modeling the whole dialogue, they trained the model on snippets taken from the dialogue turns. @xcite fined-tuned a pre-trained bart model to automatically generate summaries from doctorpatient conversations. however, they only focused on two specialties (internal medicine and primary care) and the training data only consist of hpi (history of present illness) section. @xcite proposed an algorithm called cluster2sent to generate soap notes from doctor-patient conversations. their summarization model involves both abstractive and extractive methods. @xcite studied both rnn and transformer-based sequence-to-sequence models for generating medical reports. they experimented on orthopedics data and found that sequence-to-sequence modeling is more promising. the datasets of the studies mentioned above are not made public. @xcite investigated medical conversation summarization from a chinese conversational corpus. they applied a hierarchical encodertagger model for extractive summarization to generate two types of summaries; one for problem statement and one for the treatment recommendations. several other medical conversation datasets have been recently created from chinese online health platform conversation @xcite @xcite but they do not include conversation summaries. @xcite studied the task of consultation note generation on a small set of 57 transcript-note pairs @xcite and performed a correlation study with several automatic metrics. as their focus was to compare automated metrics with human judgements, they chose models that would "produce different outputs to cover a wider range of errors" instead of attempting to benchmark sota summarization models. from the set of metrics they tested, they noted that character-based levenshtein distance, bertscore, and meteor performed best for evaluating the note generation task. ## experiments we train the summarization models on 4 nvidia tesla k80 gpus for 4 epochs. we set the learning rate to 3e-05 and regularize the training with an l2 weight decay of 0.1. we use a test set of 100 conversations and notes, randomly selected from the mts-dialog dataset. the remaining pairs are used for training (1,201 pairs) and validation (400 pairs). ## conclusion in this paper we conducted an empirical study of clinical note generation from doctor-patient encounters. this included creating a new dataset of 1,700 conversations and notes, evaluating several sota summarization models, and using multiple automated metrics and human judgements for the summarization models. our findings show that pre-finetuning transformer models plays a key role in improving factual accuracy and summary fluency, and in reducing critical fact omissions, with guided summarization improving the precision of the summary facts and reducing hallucinations at the expense of factual recall. the manual evaluation showed that the generated summaries reached a high fluency score (2.44 on a 0-3 scale) and relatively high factual f1 (0.76), but the best model still had a hallucination rate of 3% and missed 33% of the medical facts. wider research efforts are needed to design methods and models that can reduce the omission and hallucination rates as well as efficient fact-based evaluation metrics to automatically assess the factual consistency of the generated summaries. the key bottleneck limiting increased research from the nlp community on medical note generation has been the lack of generally available datasets to train and experiment on. this public release of the mts-dialog dataset will enable wider research and faster research progress on this very important and impactful nlp task. our comprehensive evaluation of multiple summarization models and evaluation metrics provides valuable baselines and references for comparison, and also shows where these systems and metrics stand on multiple dimensions of human evaluation. medical doctors today suffer from heavy documentation burden, which frequently causes physician dissatisfaction and burnout, and distracts doctors from being able to give their undivided attention to their patients. we look forward to a future where nlp research and the nlp community are able to provide doctors with tools for effective automated medical note generation, and enable them to return their focus to what they really love -providing great care for their patients.