Methods for Extracting Information from Messages from Primary Care Providers to Specialists (2020)

Abstract: Electronic consult (eConsult) systems allow specialists more flexibility to respond to referrals more efficiently, thereby increasing access in under-resourced healthcare settings like safety net systems. Understanding the usage patterns of eConsult system is an important part of improving specialist efficiency. In this work, we develop and apply classifiers to a dataset of eConsult questions from primary care providers to specialists, classifying the messages for how they were triaged by the specialist office, and the underlying type of clinical question posed by the primary care provider. We show that pre-trained transformer models are strong baselines, with improving performance from domain-specific training and shared representations.

## introduction electronic consult (econsult) systems allow primary care providers (pcps) to send short messages to specialists when they require specialist input. in many cases, a simple exchange of messages precludes the need for a standard in-person referral. econsult systems decrease wait times for a specialty appointment. @xcite an example econsult question is shown in figure 1 . in general, these questions are much shorter than, say, electronic health record texts. there is a stereotypical structure to these questions, including short histories, descriptions of the current problem, and questions about diagnosis, medication management, procedures, or other issues. when the message is received by a specialist's office, specialist reviewers in that office determine whether the patient needs to be scheduled for a specialist visit or whether the specialist may be able to answer a pcp's question directly without a visit. if a visit needs to be scheduled, the specialists decide whether it is urgent or not (in practice, whether @xmath0. she has been started on prednisone and methotrexate. she is originally @xmath1. we would appreciate an evaluation for both upper endoscopy and colonoscopy. upper endoscopy to evaluate her dysphagia and colonoscopy for malignancy screening (dermatomyositis patients are at increased risk for malignancy) figure 1 : an example econsult question patient goes to the front of the queue). because these econsult messages are unstructured, health systems do not know how they are used. automatically extracting information about the content and response to these questions can help health systems better understand the specialist needs of their pcps and guide population health management. accurately classified econsults can inform decision-making about how to allocate resources for quality improvement, additional specialist investment and medical education to best serve their patient population. in this work, we use standard support vector machine (svm)-based baselines and transformerbased pre-trained neural networks (i.e., *bert models) to classify econsult questions along two dimensions, focusing on referrals to gastroenterology and liver specialists. first, we build classifiers that attempt to learn to predict triage status (e.g., urgent or non-urgent) assigned to questions by the specialist reviewer. our goal is to use the ability (or inability) of classifiers to perform this task to understand the consistency of scheduling decisions across individual clinicians. this addresses a concern that specialist reviewers vary too much in their judgment on whether a visit is urgent or non-urgent. to do this, we performed experiments that compare classifiers trained on triage decisions of single specialist reviewers. the magnitude of inconsistency or unexplainable decisions among reviewers would inform whether these systems can consistently work as intended to reduce specialist visits safely and effectively. second, we build classifiers for the task of understanding the implicit information need that is the cause for the pcp asking the question -we call this question type. we developed an annotation scheme and annotated a sample of questions from across eight years for five question types. we then train and evaluate several classifier models, including standard architectures with problem-specific additions. our results show that triage status is difficult to learn in general, but even more difficult between reviewers, suggesting inconsistent reviewer decisions may be occurring. when classifying question type, the best-performing models are domainspecific pre-trained transformers, and that jointly training to predict different question types is the most effective technique. our best result occurs when combining domain-specific vocabularies with multi-task learning, suggesting that there is a synergistic effect between these two augmentations. bert learns contextual embeddings through pre-training on a large unlabeled corpus (including the bookscorpus (800m words) and english wikipedia (2,500m words)) via two tasks, a masked language model task and a next sentence prediction task @xcite . domain-specific bert models have been released, including biobert @xcite , which started from a bert checkpoint and extended pre-training on biomedical journal articles, scibert @xcite , which is pre-trained from scratch with its own vocabulary, and clinicalbert @xcite which started from bert checkpoints and extended pretraining using intensive care unit documents from the mimic corpus @xcite . in this work, we use vanilla bert, scibert, and two versions of clinicalbert, bio+clinical bert and bio+discharge summary bert 1 . ## discussion & conclusion within-reviewer results (diagonal of future work should explore further modeling of domain structure, including understanding question text better, but also in modeling relationships between output variables. for example, sometimes the diagnosis/management category is clear from expressions like please eval, but in other cases the request is only implicit. in these cases, the best clue is the lack of any specific procedure request. a sequential classification decision process may be able to incorporate this logic. in addition, we are continuing the annotation process, including continuing to revise guidelines to improve agreement, annotating more questions for question type in the gastroenterology specialty, and developing guidelines for additional specialties. our early results suggest that the question type classifier can still be improved with additional data, despite alreadypromising performance.