In-Domain Pre-Training Improves Clinical Note Generation from Doctor-Patient Conversations (2022)

Abstract: Summarization of doctor-patient conversations into clinical notes by medical scribes is an essential process for effective clinical care. Pre-trained transformer models have shown a great amount of success in this area, but the domain shift from standard NLP tasks to the medical domain continues to present challenges. We build upon several recent works to show that additional pre-training with in-domain medical conversations leads to performance gains for clinical summarization. In addition to conventional evaluation metrics, we also explore a clinical named entity recognition model for concept-based evaluation. Finally, we contrast long-sequence transformers with a common transformer model, BART. Overall, our findings corroborate research in non-medical domains and suggest that in-domain pre-training combined with transformers for long sequences are effective strategies for summarizing clinical encounters.

## introduction necessitated by electronic health records (ehr), physicians spend a large amount of time on documentation work @xcite , which contributes significantly to burnout @xcite , may result in lower job satisfaction @xcite , and can even increase the likelihood of errors and reduce the quality of patient care @xcite . to alleviate some of the burden on physicians, medical scribes are often used to summarize recordings or transcriptions of doctor-patient conversations into clinical notes. while this essential, yet tedious process may enable more effective clinical care, it shifts the burden onto medical scribes. furthermore, the continued reliance on human experts is expensive and only scalable to a limited degree. natural language generation models, such as the ones developed in this paper, have the potential to significantly reduce the documentation burden by providing suggested clinical notes to physicians or scribes nearly instantaneously. while still somewhat error-prone and not yet fully automated, these models are able to focus on much of the relevant information in doctor-patient conversations and distill it into a human-readable format for further review by trained medical professionals. pre-trained transformer models have revolutionized the field of natural language processing @xcite @xcite and have already been applied to various medical tasks @xcite @xcite . nonetheless, medical conversation summarization continues to present challenges due to its idiosyncrasies, foremost of which is the requirement to contain all relevant medical information rather than summarizing every part of a conversation. additionally, specialized medical vocabulary renders the use of conventional pre-trained models difficult. additional phases of in-domain pre-training have shown to be useful across a wide variety of domains and tasks @xcite , but limited work has been done on in-domain pre-training using unlabeled doctor-patient conversations. to address this, we leverage a doctor-patient conversation dataset described in section 3 to investigate two different pre-training methodologies using bart, led, and dialogled transformer models (section 4). we fine-tune all models on a subset of medical conversations with human-written summaries (section 4.2) and contrast them with a baseline of models that are not pre-trained in the medical domain using several different evaluation methods, including a transformer-based model for clinical concept extraction (sections 4.3 and 5). we show that our methods improve the performance on the medical summarization task and also evaluate the additional benefit of using models designed to work with long sequences (section 6). ## related work medical summarization recent research has devoted significant attention to the problem of summarizing medical encounters and documents in an automated fashion. @xcite describe a fully automated medical scribe using a combination of rnn and rule-based approaches to automatically recognize speech, convert it into a transcript, extract the relevant information, and convert it to a report. however, they omit any examples and results and mention that the scribe is still limited in its utility. since then, several deep learning approaches have been developed to summarize doctor-patient conversations. @xcite develop a modified pointer-generator (pg) network to summarize local snippets. furthermore, they explicitly model negation, which can cause difficulties for automatic approaches. interestingly, they report that transformer models did not work well, which is contrary to the findings in a lot of subsequent research. @xcite also use a pg model to perform the similar task of sentence alignment and snippet summarization. notably, they achieve good results using only a very small dataset. @xcite take on the challenging task of generating complete clinical summaries (soap notes) using various lstm, pg, and transformer models. they extract important utterances, cluster them, and then generate single-sentence summaries of each cluster. @xcite use a large dataset of doctor-patient conversations generated using automatic speech recognition to train a combined transformer-pg model from scratch. they are able to handle somewhat longer input because they do not rely on pre-trained transformer models. as an alternative approach to handle long conversations, @xcite use a pre-trained bart model with a two-stage chunking approach to generate summaries for a section of the clinical notes. related to the summarization of doctor-patient conversations, other research has explored the summarization of clinical notes and clinical history. @xcite use a pg network to summarize radiology findings and found that incorporating additional information in the form of background information about the patient improves the results. @xcite construct a model using a longformer encoder with a bert decoder to generate parts of discharge notes from the patient his-tory. they pre-train bert and longformer on domain-specific data and create a custom tokenizer, which yields strong results. ## dataset the dataset we use has already been described by @xcite and is composed of 83 605 clinical encounters involving doctors from many different specialties, patients, and potentially other speakers, e.g., nurses and caregivers. for each encounter, we use the de-identified doctor-patient conversation transcribed by a human. the median number of tokens in a conversation is 2040 (using the bart byte-pair encoding from @xcite , and there are a total of 203m tokens in the entire dataset. annotations are available for a subset of 1342 conversations in the form of medical summaries across internal medicine and primary care specialties. each conversation was summarized by multiple professional medical scribes into several sections, of which we only use the history of present illness (hpi) section for this paper due to its complexity and because it is usually written in complete sentences. there are an average of 17 reference summaries per doctor-patient conversation. the median conversation length in the summarization subset is 1334 tokens for a total of 2.5m tokens. the 95 th percentile corresponds to approximately 5120 tokens, which is the length limit we use for our long-sequence models. for pre-training, we exclude the entire subset of data that we have summaries for in order to avoid any data leakage and potentially biased results. in addition, we split off a random 5% of the remaining conversations as the validation dataset to monitor during pre-training. for fine-tuning, we attempt to remove poor summaries for a given conversation using an in-house rule-based system to extract medical concepts from the training summaries and only keeping the summary with the most concepts. even though this results in fewer labeled data, we have not observed a significant drop in performance. nonetheless, we keep all reference summaries for the test data. after splitting and removing extraneous summaries from training and validation data, we end up with @xcite summaries in the training, validation, and test sets, respectively. ## methods all methods are based on pre-training and/or finetuning of bart @xcite , longformer-encoder-decoder (led) @xcite , and dialogled @xcite . bart is a pre-trained encoder-decoder transformer model designed for fine-tuning on text generation tasks, such as summarization. however, it can only encode up to 1024 tokens in both its encoder and decoder, which is less than the median sequence length in our fine-tuning dataset. led and dialogled can handle significantly longer input sequences (we use 5120 and 1024 tokens for their encoders and decoders, respectively) by employing a combined global and local attention mechanism which scales linearly with sequence length. the led architecture is almost identical to that of bart except that the position embeddings of bart are copied 16 times to enable longer input. the parameters of led are initialized from bart and no additional pre-training was done. dialogled is initialized from led and further pre-trained on long dialog data using a window-based denoising task specifically designed for conversations, which results in significant improvement for long-dialog summarization. we initialize and train all of our models using the pre-trained bart, led, and dialogled models available in the hugging face transformers library @xcite . we use the corresponding tokenizers (all of which use the bart/gpt-2 bytepair encoding with a vocabulary size of 50 265), but we add additional speaker tokens, e.g., [dr]:, [pt]:, etc. we investigate both the base and large models (140m vs. 400m parameters, respectively). except for the additional position embeddings, all base models and all large models have the same number of parameters. ## experiments we establish baselines by fine-tuning base and large versions of vanilla bart, led, and dialogled models on the doctor-patient conversation summarization dataset as described in section 4.2, i.e., using the versions of those models that are pre-trained as described in their original papers. to assess whether a second round of pre-training on in-domain data is beneficial, we continue pretraining the models on our doctor-patient conversation dataset as described in section 4.1 followed by fine-tuning on the summarization dataset. for bart, window-based denoising results in a negative impact on rouge scores, so we only investigate normal text infilling denoising, whereas for led and dialogled, we consider both bartstyle text infilling and window-based denoising. the results of performing all types of evaluation described in section 4.3 on the summarization test set are shown in ## qualitative analysis in-domain pre-training across all models and pre-training objectives, rouge f1 scores always improve with additional in-domain pre-training ( overall, we find that pre-training led with the window-based denoising task leads to the strongest models in terms of rouge scores. for led-large, in-domain pre-training improves the summarization performance of doctor-patient conversations by 1.59 points for rouge-1, 1.13 points for rouge-2, and 1.10 points for rouge-l relative to the vanilla led-large baseline ( similarly, in-domain pre-training almost always improves both of our concept-based evaluation metrics with the only noticeable outlier being bartlarge. we note that we observe a slightly stronger dependence of precision and recall on sequence length ( overall, we find that pre-training dialogled with the bart-style text infilling task leads to the strongest models in terms of concept-based scores, which is contrary to the performance of dialogled when measured with rouge. this could imply that while dialogled generates extraneous text (also shown by its long generation length in comparing across denoising tasks used for pretraining, there seems to be no significant difference in terms of rouge between bart-style text infilling and window-based denoising, whereas concept-based scores improve with bart-style pre-training compared to window-based denoising. even though @xcite designed the window-based denoising task for conversation data, it seems that it is not always beneficial to use over more conventional pre-training. the most important thing is simply the process of pre-training on medical conversations itself, regardless of pretraining objective used. one benefit of using automatic metrics is that they may quantify smaller improvements which would not be as visible with small-scale human evaluation. in particular, ## conclusion we showed that in-domain pre-training improves abstractive summarization of long doctor-patient conversations into hpi notes across several models based on the bart architecture and across two different pre-training objectives. to measure the improvement, we used conventional evaluation methods like rouge and umls concept-based evaluation and also trained a neural clinical concept extraction model to better extract relevant concepts. we also demonstrated the benefit of using models that can deal with long conversations intrinsically, especially for ensuring that relevant medical concepts are present. while unlabeled doctor-patient conversations are a useful source of pre-training data, we hope to investigate additional types (e.g., clinical notes) in the future. similar research has already shown that other types of pre-training data can be very effective, e.g., pre-training on patient histories @xcite or pre-training on clinical notes for named entity recognition @xcite . additionally, we can explore combining and contrasting our holistic pre-training approach with methods that only pre-train a small amount of additional parameters @xcite . lastly, given the varying lengths of generated summaries, we are considering methods to control generation length as another future research direction @xcite .