Extracting Appointment Spans from Medical Conversations (2021)

Abstract: Extracting structured information from medical conversations can reduce the documentation burden for doctors and help patients follow through with their care plan. In this paper, we introduce a novel task of extracting appointment spans from medical conversations. We frame this task as a sequence tagging problem and focus on extracting spans for appointment reason and time. However, annotating medical conversations is expensive, time-consuming, and requires considerable domain expertise. Hence, we propose to leverage weak supervision approaches, namely incomplete supervision, inaccurate supervision, and a hybrid supervision approach and evaluate both generic and domain-specific, ELMo, and BERT embeddings using sequence tagging models. The best performing model is the domain-specific BERT variant using weak hybrid supervision and obtains an F1 score of 79.32.

## introduction increased electronic health records (ehr) documentation burden is one of the leading causes for physician burnout @xcite . although ehrs facilitate effective workflow and access to data, several studies have shown that physicians spend more than half of their workday on ehrs @xcite . this leads to decreased face time with patients and reduced work satisfaction for physicians @xcite . for these reasons, there has been growing interest in using machine learning tecniques to extract relevant information for a medical record from medical conversations @xcite . on the other hand, research shows that approximately 23% of patients do not show up for their doctor appointments @xcite . missed appointments have a large impact on hospitals' ability to provide efficient and effective services @xcite . @xcite dr: and the thing with angiogram , uh, um, um, we already scheduled, uh, the days i do it is mondays and on fridays. that's the best time for me to do the angiogram. dr: um, yeah, i, i guess i could make it monday next week. dr: or next, or friday? dr: which one is better for you? pt: um, well, it's already -today is, uh, uh, tuesday. dr: yeah, let's do it as soon as possible. pt: okay, let's do it on monday. also show that a significant number of patients miss their lab appointments. missed lab appointments can put a patient's health at risk and allow diseases to progress unnoticed @xcite . one of the main reasons for no-shows is patient forgetfulness @xcite . @xcite and @xcite show that proactive reminders through text messages, calls, and mobile applications are promising and significantly decrease the missed appointment rates. in line with the aforementioned value, appointment span extraction from medical conversations can help physicians document the care plan regarding diagnostics (dx), procedures (px), follow-ups, and referrals. it can also directly impact a patient's ability to keep their appointments. in this work, we investigate extracting the appointment reason and time spans from medical conversations as shown in figure 1 . the reason span refers to a phrase that corresponds to dx, px, follow-ups and referrals. the time span refers to a phrase that corresponds to the time of the appointment. to tackle this task, we collected a dataset for both reason and time spans and framed it as a sequence tagging problem. our contributions include: (i) defining the appointment span extraction task, (ii) describing the annotation methodology for labeling the medical conversations, (iii) investigating weak supervision approaches on sequence tagging models using both generic and domain-specific elmo and bert embeddings, and (iv) performing error analysis to gather insights for improving the performance. ## related work extracting dx, px and time expressions has been the subject of past work. tools such as clinical text analysis and knowledge extraction system (ctakes) @xcite and metama @xcite are widely used in the biomedical field to extract medical entities. both of these tools use the unified medical language system (umls) @xcite to extract and standardize medical concepts. however, umls is primarily designed for written clinical text, not for spoken medical conversations. further, research on date-time entity extraction from text is task agnostic. rule-based approaches like heideltime @xcite , and sutime (chang and manning, 2012) mainly handcraft rules to identify time expression in the text. learning-based approaches typically extract features from text and apply statistical models such as conditional random fields (crfs). while these tools perform well for generic clinical and date-time entity extraction from texts, they don't fare as well on task-specific entity extraction, where only a subset of the entities present in the text is relevant to solving the task. recently, there has been an increasing interest in medical conversations-centered applications @xcite . @xcite proposed several methods for extracting entities such as symptoms and medications and their relations. @xcite and @xcite examined the task of medication regimen extraction. while recent research in medical conversations is primarily focused on extracting symptoms and medications, we propose a new task of extracting appointment spans. our framing of this task as a sequence tagging problem is similar to @xcite ; however, they use a fully supervised approach and mainly focus on relation extraction, whereas we investigate weak supervision for appointment span extraction. moreover, we evaluate both generic and domain-specific elmo and bert models in our task. ## models in this section, we briefly describe our two models that use variants of contextualized embeddings namely, elmo @xcite and bert @xcite . ## evaluation to evaluate our models, we measure microaveraged precision, recall, and f1 of reason and time spans on the test dataset ( on the other hand, the bert variants showed a minor improvement with hybrid supervision. the bert variants consistently performed better than elmo variants, and the domain-specific pretraining has only a minor impact on bert when compared to elmo. overall, the proposed hybrid supervision approach has consistently improved performance across all model variants and the results show that augmenting the training data with inaccurate supervision can improve the performance. in order to assess performance at each span type, we chose the best performing biobert-hybrid model. for both span types precision was lower than recall ( to better understand the errors in predictions, we computed percentages of different types of errors ( complete false positives and false negatives are the significant sources of errors for both reason and time spans and our qualitative analysis suggests that these cases often happen when multiple reason phrases and time phrases are present in the utterance window, but only a subset of them are valid. because the task actually involves two different aspects, extracting reason and time mentions and spotting their confirmation clues, it may be difficult for the trained system to select exactly the confirmed reason or time mentions without explicitly modeling their relations. the ambiguity due to the oral nature of the conversations also makes it difficult to spot the confirmation clues. notably, we observe that the portion of complete false positives for the time span is significantly higher than reason spans. for example, the conversation in figure 1 discusses several options for the appointment time, but the patient finally settles for monday. the model often struggles with such cases and also extracts time mentions that are not confirmed. using spacy's ner, we find that 87% of these errors occurred when multiple time phrases are present, but not all are valid. the model may have difficulty with these cases because they amount to only 21.3% of the manually annotated time spans. further, the annotated time spans are infrequent by a factor of three than the reason spans. these reasons explain why the f1 score on time span is significantly lower than the reason span. ## conclusion in summary, we defined a novel task of extracting appointment spans from medical conversations, described our annotation methodology, and employed three weak supervision approaches to account for the limited set of annotations. our proposed hybrid weak supervision approach showed improvement across all our experiments. finally, our error analysis shows that a significant portion of the errors comes from false positives where the model has difficulty in identifying the correct span when multiple appointment reason or time mentions are present. in future work, we plan to study the data augmentation approaches as well as joint entity and relation extraction approaches to improve performance on difficult examples. we also plan to study the generalization of this work to automatic transcripts, whose transcription error rate may challenge entity detection.