Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations (2021)

Abstract: Fine-tuning pretrained models for automatically summarizing doctor-patient conversation transcripts presents many challenges: limited training data, significant domain shift, long and noisy transcripts, and high target summary variability. In this paper, we explore the feasibility of using pretrained transformer models for automatically summarizing doctor-patient conversations directly from transcripts. We show that fluent and adequate summaries can be generated with limited training data by fine-tuning BART on a specially constructed dataset. The resulting models greatly surpass the performance of an average human annotator and the quality of previous published work for the task. We evaluate multiple methods for handling long conversations, comparing them to the obvious baseline of truncating the conversation to fit the pretrained model length limit. We introduce a multistage approach that tackles the task by learning two fine-tuned models: one for summarizing conversation chunks into partial summaries, followed by one for rewriting the collection of partial summaries into a complete summary. Using a carefully chosen fine-tuning dataset, this method is shown to be effective at handling longer conversations, improving the quality of generated summaries. We conduct both an automatic evaluation (through ROUGE and two concept-based metrics focusing on medical findings) and a human evaluation (through qualitative examples from literature, assessing hallucination, generalization, fluency, and general quality of the generated summaries).

## introduction in recent years, pretrained transformer models @xcite @xcite have been responsible for 1 code is available at https://github. com/negrinho/medical_conversation_ summarization many breakthroughs in natural language processing (nlp) such as improved state-of-the-art performances for a broad range of tasks and the ability of training effective models for low-resource tasks. the demonstrated capability of transfer learning using large pretrained transformer models has led to widespread interest in leveraging these models in less standard nlp domains. medical domains provide unique challenges and great potential for practical applications (e.g., see @xcite and @xcite ). automatic generation of medical summaries from doctor-patient conversation transcripts presents several challenges such as the limited availability of supervised data, the substantial domain shift from the text typically used in pretraining, and potentially the long dialogues that exceed the length limitation of conventional transformers. additionally, the model must have both extractive (e.g., such medications being taken, medication dosage, and numeric values of test results) and abstractive (e.g., the ability to determine the onset of a symptom from multiple conversation turns) capabilities. existing work on summarization from medical dialogue transcripts has achieved only limited success, both with pretrained models and otherwise. @xcite relied on extra supervision to train a classifier to extract noteworthy utterances that are relevant to the target summary and do not handle the long conversations with their pretrained models, and their example results suffer from inferior fluency. other existing work relying extractive methods is poorly adjusted to the informal nature of dialogue and the fact that information might not be present in any single span from the conversation transcript. due to this, it has not yet been established that pretrained models are able to successfully perform automatic summarization from doctor-patient conversation transcripts. in this paper, we attempt to tackle the task of medical dialogue summarization by leveraging pre-trained transformer models. we show that bart @xcite can be fine-tuned to generate highly fluent summaries of surprisingly good quality even with a small dataset of no more than 1000 doctor-patient conversations (section 2). we overcome the input length limitations through a multistage fine-tuning approach in which the task of dialogue summarization is achieved in two steps: summarizing portions of input conversation and rewriting aggregated summaries of each portion (section 3). our approach is simple as it amounts to fine-tuning pretrained model on appropriately constructed datasets. despite its simplicity, it is effective at improving performance according to both automatic evaluation and human inspection (section 4.1-4.3) when compared to the baseline approach of simply truncating the input. we also observe good generalization of our fine-tuned models across medical domains and conversation lengths, as shown by example conversations from other papers tackling the same task such as @xcite and @xcite . these examples also show the superior quality of our generated summaries. ## dataset the dataset used in this paper is based on a collection of more than 80000 de-identified doctorpatient conversations (both audio and transcript). 1342 conversations of two major specialties: internal medicine and primary care are annotated by medical scribes using our annotation environment specifically designed for the task. the scribes listen to the conversation audio and fill in necessary information in a simulated electronic health record (ehr) system. the ehr simulator consists of 14 distinct sections such as history of present illness (hpi) and review of system (ros). we collect multiple references for each conversation, for a total of 21588 annotations. the dataset is split by conversation into train, development, and test with 939(15043), 201(3095), and 202(3450), respectively, where the values in parentheses are the number of hpi summaries in that split. additional statistics are included in appendix a.1. we choose to use only the hpi section as our training target due to several observations: first, non-hpi sections are much less frequently filled by scribes, e.g., no more than 5% of all annotations have covered ros section; second, scribes are required to write coherent paragraphs in the hpi section, whereas other sections might be structured as forms with most items being multiple choice; third, scribes are trained to cover non-hpi aspects like medication or physical examination in the hpi section if they are relevant to the "present illness" of the patient, making hpi section a good candidate for capturing most important medical findings in the conversation. each conversation in our dataset has on average 15 reference hpi summaries from different scribes. one running example of a long conversation (with more than 2200 words) and three corresponding references are showcased in appendix a.3. as can be seen from the example, different references can exhibit large variance in length and quality. for consistency, we select target reference summaries in the training set as follows: first, we leverage our rule-based system to extract medical findings from all reference summaries; then we select the reference with the most findings as target. while filtering out low-quality training summaries is expected to impact the performance of the fine-tuned models, we leave such study for future work. ## methods the methods that follow can be broken down into single-stage and multistage. all models rely on fine-tuning of pretrained bart models, the difference being how the datasets used for fine-tuning are constructed. for the single-stage approach, conversation transcripts are serialized with doctor and patient roles annotated (i.e., the encoder consumes a single sequence for the conversation) and mapped directly to the target summary. conversations longer than the transformer model length limit are simply truncated, leading to unrecoverable information loss. despite the simplicity of this approach, it works remarkably well and it serves as a strong baseline to beat. for the multistage approach, the conversation is first broken down into parts that are summarized independently by one model, and the resulting partial summaries are then aggregated and summarized into a final summary by another model. the methods that we propose in this class differ in how they break down the conversation into parts and therefore, the datasets that are used for fine-tuning their first stage model. the multistage approach is motivated by the necessity of getting around the limited length budget of pretrained models along with the belief that medical findings covered in a summary is often present locally in a contiguous set of turns between the doctor and the patient, allowing each part to be summarized independently, with a later aggregation stage of all part summaries. ## experiments we adopt rouge @xcite as our main evaluation metric. although rouge score has limited capability of capturing semantic similarities such as paraphrasing, which is common in abstractive summarization, we still consider it a useful metric for medical summarization due to restricted and highly technical vocabulary used in the medical domain. all references in dev and test set are used in automatic evaluation. to address the limitation of rouge, we also introduce an automatic concept-based evaluation metric: medically relevant findings are first extracted from both generated and reference summaries by an external nlp system, and then precision, recall, and f1 score are calculated between the two sets of findings. medical concepts are extracted via one of two systems: our in-house rule-based system and quickumls @xcite . quickumls is a python implementation of unified medical language system (umls) foot_1 that standardizes various health and biomedical vocabularies. it is publicly available, and is capable of extracting a wide scope of medical findings such as symptoms, diseases, medication and procedures. our rulebased system is a commercial system proven to be effective at capturing symptom-related findings in clinical reports. example concepts extracted from reference and generated summaries are shown in @xcite . false positive error is a major limitation of using those nlp systems, and is more severe with quickumls. we therefore implement majority voting to filter medical findings to be included in the reference set: any finding is included only if it is present in at least three human written summaries (or all of them when there are fewer references). the concept-based evaluation based on filtered findings is still susceptible to false positive errors, nevertheless, it provides an alternative to rouge as a potentially direct measure of the medical information coverage in generated summaries. such a measure aligns better with the end-user (i.e., doctors) expectation of the summary quality. we leave research on better metrics for medical summarization as future work. for automatic evaluation, we present results on the test set. results on the development set can be found in appendix a.5. as a more direct approach to quality assessment, we also conduct manual evaluation on a small sample of 10 conversations in the development set. ## related work pretrained models since the inception of bert model @xcite , the research community has come to the consensus that pretrained, transformer-based models can be effective zeroshot and few-shot learners and there is a constant interest to extend the generalizability and efficiency of such models. @xcite studied the effectiveness of transfer learning of various transformer models and proposed a unified text-totext framework for all text-based language tasks. @xcite and its earlier versions @xcite showed that it is possible to elicit specific information from the model by providing an appropriate query, or "priming the model". in our work, this is effectively done in annotating each utterance with the corresponding speaker role and breaking the conversation in chunks containing information about the start of the conversation. ## conclusion in this paper, we show the feasibility of summarizing doctor-patient conversation directly from transcripts without an extractive component. we fine-tune various pretrained transformer models for the task of generating the history of present illness (hpi) section in a typical medical report from the transcript and achieve surprisingly good performance through pretrained bart models. we propose a simple yet general two-stage fine-tuning approach for handling the input length limitation of transformer models: first, a conversation is broken into smaller portions that fit within the length budget of the model and a summarizer is trained on these portions to generate partial summaries; second, we aggregate the generated partial summaries and use them for training a second summarizer to complete the summarization. we show that this approach can help the model pick up medical findings dispersed across long conversations and reduce hallucination compared to single stage fine-tuning. to the best of our knowledge, our work is the first to show the feasibility of generating fluent summaries directly from doctor-patient conversation transcripts. of practical concern for medical applications, hallucination and missing information in our generated summaries can be serious problems, nevertheless, we believe our results are encouraging, especially for assisting a scribe in a human-in-the-loop system. we also plan as future work to further explore this task in the aspect of multiple reference summarization and better evaluation metrics that align with quality assessment in the medical domain. ethical considerations medical conversation summarization inevitably deals with medical data which could potentially contain sensitive information about patients and doctors alike. careful deidentification for removing all sensitive and identifiable information in the input data is an important tool for privacy protection. we ensured that our data went through a similar process to not reveal any sensitive information (age, name, home address, etc.) about all people involved or mentioned in the conversation. the same de-identified data is also presented to scribes during annotation to ensure no leakage of sensitive information. no information about gender, ethnicity or other discriminating factors are used as a part of our proposed method. the intended use of our method is for designing an automatic summarization system aimed at reducing physician and scribe burnout due to the burdersome documentation process required for each medical encounter. the most natural application of this technology is not as a replacement for a human scribe, but as an assistant to one. by providing tools that aid a human scribe one can mitigate much of the risk of system failures, such as hallucination. nonetheless, continued work is required in this area to ensure that both privacy and data accuracy are preserved.