Generating Medical Reports from Patient-Doctor Conversations Using Sequence-to-Sequence Models (2020)

Abstract: We discuss automatic creation of medical reports from ASR-generated patient-doctor conversational transcripts using an end-to-end neural summarization approach. We explore both recurrent neural network (RNN) and Transformer-based sequence-to-sequence architectures for summarizing medical conversations. We have incorporated enhancements to these architectures, such as the pointer-generator network that facilitates copying parts of the conversations to the reports, and a hierarchical RNN encoder that makes RNN training three times faster with long inputs. A comparison of the relative improvements from the different model architectures over an oracle extractive baseline is provided on a dataset of 800k orthopedic encounters. Consistent with observations in literature for machine translation and related tasks, we find the Transformer models outperform RNN in accuracy, while taking less than half the time to train. Significantly large wins over a strong oracle baseline indicate that sequence-to-sequence modeling is a promising approach for automatic generation of medical reports, in the presence of data at scale.

## introduction there has been an increase in medical documentation requirements over the years owing to increased regulatory requirements, compliance for insurance reimbursement, caution over litigation risk, and more recently towards increased patient participation. according to a study on 57 u.s. physicians, for every hour with a patient, a physician takes an additional hour of personal time doing clerical work @xcite . increased documentation burden has been identified as one of the main contributing factors for physician burnout @xcite . in another, larger study, u.s. physicians who used electronic health records (ehrs) or computerized physician order entry (cpoe) were found to be less satisfied with the time spent on administrative work @xcite . increased physician burnout not only affects the health and well-being of the physicians, it can also lead to increased medical errors, increased job turnover, reduced productivity, and reduced quality of patient care @xcite . factors related to physician burnout and its consequences have been studied in detail in the literature @xcite . use of automatic speech recognition (asr) to dictate medical documentation has contributed significantly to the efficiency of physicians in creating narrative reports @xcite . however the content of the report has already been discussed with the patient during the encounter. medication list and orders entered into the ehrs are also discussed with the patient. in other words, creation of medical documentation by the physician may be viewed as a redundant task given that the content is already discussed with the patient. there has been a surge in research on automatic creation of medical documentation from patientdoctor conversations. a lot of it is focused on extracting medical information and facts from the patient-doctor conversation @xcite . this could involve extracting clinical standard codes @xcite , clinical entities such as symptoms, medications, and their properties @xcite , or medical regimen (selvaraj and konam, 2020). this extracted information could then be used to generate a report @xcite . such information extraction systems require creating an annotated conversation corpus @xcite . for example, the nlp pipeline described by @xcite first extracts knowledge from an asr transcript and then generates the report. the knowledge extraction consists of tagging speaker turns and sentences with certain classes using rnnbased models, and using techniques such as string matching, regular expressions, and data-driven supervised and unsupervised approaches to extract information from the tagged sentences. this is followed by data-driven templates and finite state grammars for report generation. we take a different approach where the problem is cast as translation (source language is conversational in nature and target language is clinical) and summarization (input contains redundant and irrelevant information, and target is a concise and precise note) at the same time. given recent advances in neural transduction technology @xcite @xcite , we explore the end-to-end paradigm for generating medical reports from asr transcripts. this eliminates the need for annotated corpora that are required for training intermediate processing steps. as a result this approach is scalable across various medical specialties. sequence-to-sequence models have been used for summarizing radiology notes into the short impressions section, possibly incorporating also other domain-specific information @xcite . in contrast, our system creates a report directly from the conversation transcript. disadvantages of the end-to-end approach include that it limits the ability to inject prior knowledge and audit system output, and may potentially result in inferior performance. ## dataset we use data consisting of ambulatory orthopedic surgery encounters. speaker-diarized conversation transcripts corresponding to the audio files were obtained using an automatic speech recognizer. the reports for orthopedic surgery are organized under four sections-history of present illness (hpi), physical examination (pe), assessment and plan (ap), and diagnostic imaging results (res). the hpi section captures the reason for visit, and the relevant clinical and social history. the pe section captures both normal and abnormal findings from a physical examination. the res section outlines impressions from diagnostics images such as x-ray and ct scans. finally, the ap section captures the assessment by the doctor and treatment plan e.g. medications, physical therapy etc. experimental results are reported on a dataset that consists of around 800k encounters from 280 doctors. the dataset is partitioned chronologically (date of collection) into train, validation and evaluation partitions. the evaluation partition includes 4,000 encounters from 80 doctors. the doctors present in the evaluation set are present in the train set. since the models do not require supervision outside the workflow, this paradigm is scalable, though future work will assess generalization to unseen doctors. we only use non-empty examples for training and evaluation. the res section is empty in about 50 % of the examples. ## modeling we use neural sequence-to-sequence models for summarizing the doctor-patient conversations. the input to the model is a sequence of tokens generated by the speech recognizer. the medical reports consist of four sections, and we produce each section using a separate sequence-to-sequence model. the task closely resembles machine translation, so we use models that are similar to neural machine translation models. there are, however, several differences from a typical machine translation task: the models that we use are based on the encoderdecoder architecture that is well known from neural machine translation @xcite . all the models rely on attention @xcite . the encoder creates context-dependent representations of the input tokens, and the decoder produces the next-token probability from those representations. during inference the output is generated autoregressively using the next-token probabilities. very long sequences increase the memory usage and training time, and make it more difficult to learn the model parameters. we truncate the source sequences to 2,000 tokens and the target sequences to 500 tokens during training. 10 % of the source sequences and 0.1 % of the target sequences were above this threshold. during inference we truncate the inputs to 3,000 tokens. only 4 % of the test examples were originally longer than this limit. in this work we compare models that are based on recurrent neural networks and models based on transformer @xcite . the following sections describe these models and the enhancements that we have implemented. ## experiments we train the models on azure cloud using nvidia v100 gpus. each training job is distributed to 8 gpus. we use data-parallel training, i.e. each gpu processes their share of the mini-batch and then the gradients are averaged over the gpu devices. the batch size is set to a maximum of 7,000 source tokens per gpu. nvidia nccl library is used to perform the communication efficiently. we use a vocabulary consisting of the 10k most frequent words. the same vocabulary is shared between the source and target tokens. we use @xcite ) with the rnn models, while adam (kingma and ba, 2015) is found to perform better with the transformer models. we train the models a maximum of 400k steps, excepting res section models, which are trained until 200k steps due to their fewer examples and shorter targets. this corresponds to approximately 25 epochs on res section and 30 epochs on other sections. during this time we observe that training has practically converged and training longer would not provide significant benefit. we also limit individual model training to one week as a cost control. improved performance is obtained via averaging model parameter from 8 checkpoints, with interval length as a function of total training steps. where helpful, we use a cyclical learning rate schedule, with the cycle length set to the checkpoint saving interval, so that the saved checkpoints would correspond to the minimums of the learning rate schedule @xcite . ## conclusions in this paper we compared rnn and transformerbased sequence-to-sequence architectures for medical report generation from patient-doctor conversations. this study demonstrates the ability of sequence-to-sequence models, in particular transformer, to not only extract relevant clinical conversation excerpts, but abstractively summarize in a relatively fluent and factually correct medical report. especially when working within compute and time budgets, transformer is superior to traditional rnn-based models, and scalable to large datasets. visual inspection showed that commonly occurring problems in the generated reports included repeated sentences and hallucinated clinically consistent sentences unfounded by the conversations. minimally a human would need to be in the loop to verify or correct these machine-generated reports. future work includes comparing end-to-end approaches with a pipeline of clinical information extraction and natural language generation methods.