Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structures. (2020)

Abstract: Understanding a medical conversation between a patient and a physician poses unique natural language understanding challenge since it combines elements of standard open-ended conversation with very domain-specific elements that require expertise and medical knowledge. Summarization of medical conversations is a particularly important aspect of medical conversation understanding since it addresses a very real need in medical practice: capturing the most important aspects of a medical encounter so that they can be used for medical decision making and subsequent follow ups. In this paper we present a novel approach to medical conversation summarization that leverages the unique and independent local structures created when gathering a patient’s medical history. Our approach is a variation of the pointer generator network where we introduce a penalty on the generator distribution, and we explicitly model negations. The model also captures important properties of medical conversations such as medical knowledge coming from standardized medical ontologies better than when those concepts are introduced explicitly. Through evaluation by doctors, we show that our approach is preferred on twice the number of summaries to the baseline pointer generator model and captures most or all of the information in 80% of the conversations making it a realistic alternative to costly manual summarization by medical experts.

## introduction telemedicine is a rapidly growing medium of interaction with the healthcare system @xcite . with the covid-19 pandemic limiting in person medical visits, healthcare systems are seeing greater than 100% increase in virtual urgent care visits and greater than 4000% increase in dr: you mentioned having a cough for 2 days and a fever since last night along with being short of breath. is that correct? pt: yes , correct dr: i appreciate your concern for preventing spread. do you feel like you are unable to move around as usual? pt: i'm definitely weaker and low energy the fever went down to 99 this morning dr: have you taken any medications or tried anything else to help you with your symptoms? pt: lots of fluids and vitamin c. lozenges to minimize coughing dr: do you have any medical conditions or have you been on any medications pt: no, none dr: alright. when you had a fever, did you take medicine like tylenol to bring the fever down? pt: i didn't ## model we are interested in a model that has two main properties. first, it encourages copying from the snippet to preserve the integrity of the symptoms and medical issues being discussed. second, it can handle out-of-vocabulary terms, such as medically relevant terms, that are used by patients and doctors. the pointer generator @xcite is naturally suited as they imbibe these properties by providing a hybrid of extractive and abstractive summarization, with more emphasis on extraction @xcite . we use pointer generator as the base model to build upon and encode medical conversation specific properties. ## evaluation we evaluate models using automated metrics and manual evaluation from doctors. multiple studies have shown that automated metrics in nlp do not always correlate well to human judgments as they may not fully capture coherent sentence structure and semantics @xcite . since medical dialogue summarization would be used to assist health care, it is important for doctors to evaluate the quality of the output. ## dataset construction we collected a random subset of dialogue of 25,000 conversations from a telemedicine platform. we split the dialogue into a series of local dialogue snippets using a simple heuristic: the turns between two subsequent question by the physician corresponds to a snippet. the length of these snip-pets ranged anywhere from two turns (a physician question and patient response) to ten turns. we had medical doctors summarize a random sample of 3000 snippets. these are the same doctors who practice on the same telemedicine platform. the doctors were asked to summarize the sections as they would for a typical clinical note by including all the relevant information. further if the summary included a negated medical term, eg. "doesn't have fever", the doctors were asked to use a [no] token in front of that particular sentence in the summary. if a local snippet did not contain any relevant information they were excluded from annotations. for example in the beginning or end of conversations there may be turns that are purely greetings and not part of the patient history taking process, or purely educational in nature. at the end of the labeling, we used the 1690 number of local snippets that the doctors labeled as containing pertinent information for history gathering. we used 1365 as the training set, 158 as a validation set and 167 as a held out test set. the test set was made sure to be from distinct conversations that were taken from a different date range on the platform compared to the training or validation sets. this was done to ensure that different local snippets from a certain conversation weren't part of the training and test sets. the data was preprocessed by removing the names of the actors in the dialogue ("doctor", "patient") and concatenating all the turns within a snippet together. training details: all the models use a vocabulary size of 50k with 128 dimensional embeddings and 256 dimensional hidden states. the training parameters followed @xcite with a learning rate of 0.15 and adagrad as the optimizer. the coverage mechanism as described in @xcite was used for all our models. models were first pretrained on the cnn-daily mail corpus and finetuned on conversational data from our in-house chat-based telehealth platform ( § 5). pretraining took approximately 2 days on a single nvidia titan xp gpu and finetuning took under 2 hours. the concept and negation attention modifications added 512 parameters each to the base pointer generator model with coverage. for details on hyperparameters and validation results see supplement. ## conclusions in this paper, we presented a novel approach to medical conversation summarization. this is an important application for text summarization since medical professionals rely on good conversation summarizations for medical decision making and follow up. medical conversations, however, have traditionally posed a challenge for vanilla machine learning approaches because of the importance of domain knowledge and syntactic nuances such as negation. we extend a deep learning approach, pointer generator networks, and show that for domains like medicine where integrity of the source is critical, encouraging copying in the learning process produces the best model (2m-pgen) on human evaluation. this approach represents a viable alternative to human summarization since experts report that up to 80% of the relevant information is present in 2m-pgen summaries with only 5% of summaries containing no relevant information. even if the system implementing this approach could not operate completely automated, it is clear that it could speed up the summarization process by reducing the amount of human intervention needed. for future work, we would like to see if our findings generalize well on other datasets and other domains. particularly, we would like to see if the anecdotal evidence that explicit negation modeling matters, can be captured by the metrics or the expert human evaluation.