In recent years, there has been a broad proliferation of online symptom checkers and medical triage solutions as many patients seek web-based medical guidance prior to engaging with their health care provider (White and Horvitz, 2009; Semigran et al., 2015; Chambers et al., 2019; Dunn, 2020) . A typical symptom checker functions by asking patients a sequence of questions and generating a set of possible conditions and treatment options. Symptom checkers vary by their operational range -some target specific populations (e.g. pediatrics) or disease states (e.g. influenza), while others have broader scopes (e.g. low acuity primary care) (Kellermann et al., 2010; Price et al., 2013; Buo; Ada; Inf; Bab; Med) .
The need for high quality symptom checkers has grown with the COVID-19 pandemic of 2020, which caused a significant decrease in utilization of primary and urgent care (Gelburd, 2020) .
Suddenly, many millions of people are seeking diagnostic assistance online and up to 90% of primary care visits have shifted to virtual care. The transition to (in some cases) majority virtual care represents an opportunity for greater symptom checker usage and an imperative for tight integration into the health care operational workflow. The EMR note is the documentation that is automatically, and dynamically, generated by the data in the EMR activity. Positive symptoms are in red.
Symptom checkers with actionable outcomes can provide a host of benefits. Computer assisted triage of low acuity patients can decrease clinician patient load, while decision support for ordering of appropriate laboratory testing (including COVID-19 testing, (Greenhalgh et al., 2020; Judson et al., 2020) ), medications, and imaging studies, can eliminate unnecessary visits. With advanced decision support and documentation assistance, patient visits themselves are enhanced as the clerical load (e.g. documentation, coding, ordering) is decreased.
Obtaining maximal benefit from symptom checkers requires a tight integration with the Electronic Medical Record (EMR). An integrated system can utilize knowledge gleaned directly from the patient and can also leverage historically relevant patient medical information. In addition, symptom checker output that is manipulable and represented in appropriate locations within the EMR can help facilitate clinical workflow and improve quality of care.
In this paper we present SmartTriage, a symptom checker that is tightly integrated with the EMR and which has a broad scope covering 337 presenting chief complaints. Figures 1 and 2 depict our main EMR integration, with a specially built activity that allows for discrete manipulation of each reported patient response ( Figure 1a ) and clinical decision support (CDS) elements ( Figure 2 ). Figure 1b shows what our documentation assistance looks like, with text written directly into the current progress note. The patient view of smart triage can be seen in Figure B .2.
We use machine learning to bootstrap question sequencing and decision support predictions through training on two novel data sets. The first data set contains nearly one million patient generated free-text reasons-for-visit entries, each of which is mapped to a discrete chief complaint. The second data set is a collection of 25 million patient encounters including progress notes as well as discrete diagnoses and orders.
Three machine learning models enable SmartTriage. Our chief complaint extraction model (CC model) uses a fine-tuned bi-directional transformer neural network (BERT) to process patient freetext reason-for-visit entries (Devlin et al., 2019) . A collection of over 3000 random forest models were trained to produce highly informative question sequences (QS model). Finally, a feed-forward network was created to produce clinical assessments (Assessment model), taking into account all historical data and all questions answered. Of note, the CC and the Assessment models use a long short-term memory (LSTM) based patient history representation (Hochreiter and Schmidhuber, 1997) . SmartTriage is currently undergoing a pilot as a pre-visit symptom checker at a single urgent care facility, with a second pilot planned for preceding virtual visits (phone, video) before the end of 2020. Details about production integration can be found in Appendix B.
SmartTriage is built on historical EMR data from two novel data sets. The first is based on a collection of patient reasons-for-visit entries and the second is based on a collection of primary care encounters. Figure A .1 shows how the data sets were acquired.
When a patient schedules an appointment online they are prompted to enter a free-text reasonfor-visit. At the time of the encounter, a discrete chief complaint is chosen by the intake personnel. The combination of patient entered text, the discrete chief complaint(s), and 1 year of patient history from the EMR form the Patient-Reason for-Visit data set (see Appendix A.1).
During a primary care encounter a progress note is typically generated containing the patient's history of present illness, past medical history, a review of systems, a physical exam, and an assessment and plan. The encounter may also result in a set of discretely coded outcomes such as ICD-10-CM codes for diagnoses, medication prescriptions, and laboratory or imaging orders. We use an in-house developed natural language processing (NLP) engine to extract Unified Medical Language System (UMLS) clinical concepts from progress notes (Bodenreider, 2004) . The extracted discrete concepts, encounter outcomes, and 1 year of patient history from the EMR form the Encounters data set (see Appendix A.2).
Clinical concepts from the UMLS are by their nature very detailed and clinically focused (Bodenreider, 2004) . The level of detail often implies that multiple concepts exist to capture the same idea in different contexts. Figure 3 shows three concepts that describe gastric contents backing up into the esophagus -"Heartburn" as the sign or symptom, "Gastroesophageal reflux" as the finding on exam, and "reflux" as a pathological process in general. Although there are context clues in medical notes, clinicians will often use these phrases or variants to imply the same idea, and our NLP engine will produce one or more of these concepts for the same span of clinical text. In order to prepare a high quality training data set for our QS and Assessment models, we employed many man-hours of clinical review to collapse all of the items found by our NLP engine into meaningful "main idea" groups. In large part, the groups contained perfectly synonymous concepts; in some cases, however, we grouped items that were not perfectly synonymous but implied the same concept. Our pilot system contains more than 3K distinct SmartTriage Content Management System (ST-CMS) concepts.
SmartTriage is both patient and provider facing, and thus we had the dual challenge of authoring our grouped concepts such that they would be understandable to patients as questions and to providers as clinical concepts. In addition, because we needed to make the concepts digestible into the EMR and into our question-answers system, each concept had to be tagged with metadata describing where it should go in the clinical note and how it should be displayed to the patients. Figure C .1 shows the system we built to manage all of our content, including concept groupings, patient facing text, provider facing text, and question metadata.
There are three machine-learning-based components in SmartTriage. In order of operation, the Chief Complaint (CC) model takes patient history as well as a patient entered free-text reasonfor-visit and generates predictions for the discrete chief complaint. The Question Sequencing (QS) model uses the discrete chief complaint, patient history, and patient responses, to interactively ask a series of questions with discrete response choices. Finally, the Assessment model employs all of the collected information to generate predictions for diagnoses as well as medication, laboratory, and imaging orders.
Architecture A feed-forward neural network was constructed to combine pre-trained free-text embeddings with patient age (discretized into age groups), biological sex, and patient medical history (see Figure D .1). As multiple discrete chief complaints can be extracted from a given patient-generated reason-for-visit, the output of the model is a one-vs-rest classification for 337 chief complaints.
Patient free-text was embedded using a pre-trained language model. We fine-tuned our Chief Complaint (CC) BERT on the training portion of the CC data set, using the masked language model prediction task (Valmianski et al., 2019) . To generate BERT-based embeddings, we mean-pooled the last 4 layers, yielding an embedding size of 3,072.
The CC model utilizes patients medical history from encounters over the preceding 365 days. The history is composed of 4 different channels: previous diagnoses (1,497 distinct values), prescriptions (1,278), procedures (758), and chief complaints (954). Data from each historic encounter were placed in the four channels and channels with no relevant information were filtered out. Each channel contained the last 8 encounters with relevant information sorted in ascending order by the encounter date. We zero padded channels in cases where there were data from fewer than eight encounters.
The success of LSTM based patient history learning (e.g. Pham et al. (2017) ; Rajkomar et al. (2018) ) led us to incorporate medical history and demographic data via directly learned embeddings. Each one of the medical history channel embeddings is sized to 50 and processed through an LSTM layer (see Figure D .3). The output of the recurrent layers and the demographic embeddings were then combined with the pre-trained free-text embeddings.
The final embeddings were fed into a dense ReLU layer before being passed to the output layer. The output layer consisted of one-vs-rest logistic regressions for 337 discrete chief complaints. Dropout was used after the embedding layers and after the ReLU layer to regularize the network. Figure 4 : Impact of adding historical information to the chief complaint model for common patient specified reasons for visit
Medical history contribution We examined the effect of including patient medical history on CC model performance by training a baseline model without the medical history input and embedding layers. The aggregate effect was modest -the network containing medical history information achieved 0.442 Precision-Recall area under the curve (PR-AUC), while the network without history achieved 0.436 PR-AUC. The impact of history was skewed, however, which was revealed by looking at PR-AUC changes achieved for complaints containing free-text terms with prevalence greater than 1% (72 terms total). For each of these terms we bucketed chief complaints containing the relevant term and compared the PR-AUC of both models (see Figure 4 ). For 56 of the groups the model with history over-performed, and for the remaining 16 groups it underperformed. The highest impact of patient history was seen in free-text reasons-for-visit containing words relating to previous encounters, such as "annual", "visit", "review", and "follow".
Question sequencing occurs after the chief complaint is established. By asking a series of structured and ML-derived questions, SmartTriage captures patient data that can be used to help generate the progress note. Patient responses are also fed as inputs to the assessment model. All questions and answers displayed to the patient are mapped to ST-CMS concepts (see Section 3). Questions may be multi-select, single-select, yes/no, drop-down (for duration questions), and free-text. Each answer has an assertion associated with it that is typically drawn from {"certain", "absent", "unsure"}. For duration and severity questions the assertions are ordinal (number of days for duration and integer-scale between [0,10] for severity). Individual questions in SmartTriage are required to be interpretable and appropriate. Interpretable means that we can explain why each question was asked based on previous answers and patient medical history. Appropriate means that SmartTriage should not ask questions whose answers could be inferred either from patient history or from the set of previous patient responses. To satisfy the appropriateness requirement we use a combination of rules and machine learning, both of which have available information on the chief complaint, previously answered questions, patient demographics, and 6-months worth of patient medical history (previous chief complaints, diagnoses, laboratory results, and medications; note that this is different from the CC and the Assessment models that use 1 year worth of patient medical history).
Architecture The full QS algorithm is described in Algorithm 1. At every step the asserted ST-CMS concepts (features) are used to infer additional asserted concepts via a rules engine. For the full set of 337 covered chief complaints, the initial questions are stereotyped to collect a thorough history of present illness (HPI) with relatively minimal hard-coded branching to cover gender differences, timing variations, problem location, etc. For a subset of 269 chief complaints, additional ML questions may be generated after the HPI questions are exhausted. The ML questions are subject to additional "fixer" rules, and can extend the question sequence up to a set number of total questions (23 for our pilot).
Input : Feature list X, min count m Output: Next question q or None
Select question candidates and compute their weights
Rules-derived questions The function NextQuestionRule returns a rule-based ST-CMS concept corresponding to the next question that needs to be asked. If all the stereotyped HPI questions have already been asked then NextQuestionRule returns None and the ML questions are then asked. NextQuestionRule can also return a special EndEarly concept which prevents additional ML questions and ends the interaction.
Training ML question sequencing We use the encounter data set (Section A.2) to train the ML question sequencing model. For maximal interpretability, appropriateness, and ability to handle large feature spaces (> 1000 features per model) we chose to train a random forest model for each combination of chief complaint, age-bin, and biological sex. The models were trained independently using ST-CMS concepts and assertions (derived from NLP on progress notes) as features and relevant ICD-10-CM diagnoses as labels. The random forest models are trained with 100 estimators, feature sub-sampling of 10%, and a max depth of 15. Traverse e using features x ∈ X until reaching node n that is either a leaf node or cannot be split using X 4 if n is not a leaf node then 5 Let q be question to obtain feature x that splits n
Data cleaning To obtain highly relevant training data both concepts and ICD-10-CM diagnoses were filtered. For ICD-10-CM diagnoses we calculated the Bayesian lift that a diagnosis t would appear given the chief-complaint c, age-bin a, and biological sex s i.e. lift[t|c, a, s] = P[t|c,a,s] P [t] . For each encounter, target values with lift[t|c, a, s] ≤ 2 were removed. Overall, for every cohort of chief complaint, age-bin, and sex, we filtered out diagnoses and concepts that appeared fewer than 100 times. We further limited the input to the 1000 most-common concepts, and limited the prediction targets to the 50 most-common diagnoses (note that the assessment model, which is used for the final differential diagnosis prediction, does not have this limit).
Selecting next ML question At inference time we take a vote from the random forest using Algorithm 2. For every tree in the forest we traverse the nodes until we find a ST-CMS concept for which there is no assertion and extract the relevant question q i . For each question we calculate a weight, w q i that is the number of trees that are split on a concept asked by q i .
Rule-based "fixers" for ML questions We apply two rules to ML question results (see Algorithm 1). The PrereqRule function looks up for a given concept if another question should be asked first (for example, for chief complaint "foot problem" before asking "How long have you had ankle pain?" we check for the presence of "ankle pain"). If the answer to the prerequisite question is asserted as "absent" then the weight of the target question is set to 0. If the answer to the prerequisite question is asserted as "certain" then the question will not be modified. If the Figure 5 : Impact of adding historical information to question sequencing model by age and biological sex prerequisite question has not been answered then an additional boolean flag decides if the prerequisite question is first asked (e.g. question "How long have you had ankle pain?" is replaced by "Do you have ankle pain?") or if the question can be asked without changes. We then sum the weights of all questions that are mapped to the same prerequisite. Finally, we have an additional ToBeExcludedRule function, which removes certain ML questions that are inappropriate given the chief complaint (e.g. some insomnia sequences that include sleep apnea may result in questions related to coughing that should not be asked). The highest weighted question at the conclusion of all ML and rule-based steps is then asked.
Medical history contribution We trained the random forest models with and without patient medical history. Both models achieved very similar ROC-AUC ( 3% aggregate improvement with history), but the sequences trained on models with patient medical history resulted in fewer total questions ( Figure 5 ). The inclusion of historical data decreased the number of questions that needed to be asked by up to 20% with the effect being larger for older adults.
The Assessment model is the final ML model in the SmartTriage workflow, and it uses all of the collected information to predict the encounter diagnoses (1,240 unique values) as well as medications (122 unique values), laboratory (291 unique values), and imaging (208 unique values) orders.
Architecture Inputs to the Assessment model are comprised of the discrete chief complaint, patient demographic data, answers from the question sequence, and patient medical history. The model itself is a feed-forward neural network ( Figure D. 2) with an LSTM based patient history embedding ( Figure D.3) . Each of the inputs is embedded separately, and then the embeddings are concatenated together. The combined embbeding is passed through two ReLU dense layers of size 1024. The representation returned from the second layer is used by the four one-vs-rest logistic regression output layers.
Target cleaning For each target value t and chief-complaint c, the Bayesian lift of t appearing given c were calculated, i.e. lift[t|c] = P[t|c] P [t] . Targets with a value lift[t|c] ≤ 2 were filtered out. Loss weighting Loss for each output layer was weighted differently by performing a random search, limited to 10 combinations, over a 5% sample of the data, where each of the loss terms (L diag , L med , L lab , and L img ) was weighted by w ∈ {0.5, 0.9, 1.0, 2.0, 3.0}. The different weight combinations were sorted by the PR-AUC of all four output types, and the combination with highest rank in its worst performing output type was selected. The highest performing weighting fucntion found was L = 3 · L diag + L med + 0.9 · L lab + 0.9 · L img .
Medical history contribution The impact of patient medical history on the quality of the Assessment model is depicted in Table 1 . To generate these results, models with and without history embeddings were trained. Across the board, the assessment model incorporating patient medical history information outperformed the model lacking such information, with a lift of 7.6%, 14.3%, 8.2% , and 41.7% in PR-AUC for diagnoses, medications, labs, and imaging respectively.
Here we have demonstrated SmartTriage, an automated system for engaging with patients virtually and for providing documentation and decision support to their caregivers. Because SmartTriage was fully developed in-house, we were able to address two large short-comings of conventional symptoms checkers: (1) inability to utilize patient historical information and (2) lack of integration with an EMR. SmartTriage currently encodes up to a year of historical patient medical data, which includes demographic information, previous encounters, diagnoses, medications, procedures, laboratory results, and more. This information is used to avoid asking questions whose answers are already known (e.g. age, previous diagnoses), and uplifts the performance of all ML components utilized in SmartTriage. The ML components include the natural language understanding model for obtaining a chief complaint, the question sequencing model, and the assessment model for providing decision support.
Integrating with an EMR is crucial as it allows direct feedback from health care providers, and helps avoid unnecessary repetition for patients and gaps in continuity of care. SmartTriage's EMR integration provides assurance that there is an easily understood record of the patient's interactions with the system while supporting clinicians in their task of generating documentation, orders, and diagnoses.
To create our pilot version of SmartTriage, we bootstrapped the system by using historical clinical encounters as simulated patient interactions, with clinician documented concepts in place of actual patient provided answers. While we were able to obtain good decision support performance and clinically relevant question sequences, we recognize that the responses provided by patients may differ significantly from those in our training set in terms of interpretation (how the patient interprets the authored question) and overall question density (how many positive and negative examples are generated compared to the statistics found with NLP on clinical notes). We plan to continually retrain the system utilizing patient provided responses as more and more patients go through SmartTriage.
As the name implies, SmartTriage is meant to be a system for triaging patients. Our initial pilots and short-term roll-outs are an intermediate step to collecting the necessary ingredients for making high-quality triage decisions possible. By providing questionnaires to patients who have a forthcoming visit with a health care provider, we ensure that their data are appropriately reviewed, allowing us to learn how SmartTriage's decision support performs in real encounter situations. Furthermore, as part of the EMR activity we developed, we ask caregivers to indicate what the most appropriate venue of care would have been for the visit, including self-care, a phone or video visit, an in-person primary care visit, urgent care, or the emergency department. The triage and decision support feedback will allow us to continually improve the SmartTriage system and ultimately position it as a tool for getting the right kind of care for the right kind of patient at the right time.
Appendix A. Data sets Figure A .1: Diagram depicting how the patient reason-for-visit and patient encounter data sets are acquired. For the patient reason-for-visit data set, a free-text reason is entered when the appointment is scheduled. At the time of the encounter intake personnel choose a discrete chief complaint from a structured list. For the patient encounter data set, we combine discrete outcomes coded by the physician with NLP extracted concepts from clinical progress notes. Both data sets are further bundled with patient history elements including previous chief complaints, diagnoses, medications, labs, and orders
We developed a Patient-Reason-for-Visit data set consisting of patient-generated reason-for-visit free-text entry, discrete chief complaints, demographic information, and patient historical information. The data set was constructed similarly to (Valmianski et al., 2019) but with the addition of one year of historical information for each encounter. Patient medical history contained four information types: previous chief-complaints (954 unique values), medications (1,276), diagnoses (1,497), and procedures (758). Medical history was limited to encounters that took place at most 365 days prior to the current visit. For each information source, the last 8 encounters containing relevant information were kept. The mean number of historic encounters with previous chief-complaints information is 5.8, where the mean number of chief-complaints per historic encounter was 1.5. For historic diagnoses the mean number of encounters with relevant information was 6.4, and the mean number of diagnoses per encounter was 2. For medications information there were 3.6 encounters on average, where each encounter contained 2 medications on average. For procedures the mean number of historic encounters was 4.2, with 1.6 relevant procedures on average.
Of the 1,296,210 encounters retrieved, there were 1,139,585 unique text entries that were assigned 2,050,851 chief complaints (of which 2,362 were unique). A subset of encounters had very generic patient text or non-specific chief complaints, and we used simple string matching to remove such examples. The resulting data set consisted of 907,170 encounters for 586,254 members. There were 708,782 unique free-text entries mapped to a total of 1,250,749 discrete chief complaints, 1,701 of which were unique. For our model development, we chose to classify a subset of 337 high frequency chief complaints. The median reason for visit text was 34 characters long, with a median word count of 6 (4 without stop words). For the reasons-for-visit entries with additional free-text comments added by intake personnel, we cropped out comments to enforce a 50 character limit.
Among the patients in the final chief complaint data set, 335,476 (57%) were female while 250,778 (43%) were male. The average age was 49 years old. A full breakdown by age and sex can be found in Tables A.1 and A.2.
The data set was split into a training set of 816,357 encounters and 90,813 test encounters such that no sentences in the test set had an exact match in the training set. To create the content for our platform we employed a strategy that leveraged our organization's large patient membership. The primary input knowledge source was a set of 25,047,535 historical primary care encounters with chief complaints, clinical notes and patient demographics. Our output data source consisted of the resulting ICD10 diagnoses, laboratory orders, imaging studies, and medications that were associated with the encounters.
The clinical notes we extracted are largely unstructured text documents that contain roughly the same general categories of information -why the patient was being seen, a brief medical history, a review of systems and symptomatology, a focused physical exam, an assessment, and a plan. To access these critical elements of the patient's presentation we processed all of the notes to extract clinical concepts using an in-house developed NLP engine. The engine incorporates, among other sources, the UMLS (Unified Medical Language System) terminology. Each note yielded between 10-200 (or more) medical history items, symptoms, physical exam findings, and some elements of patient social and family history. The NLP engine assigns every item a probability that it is present (positive) or absent (negative) and identifies the section in the note that the item came from. For the purposes of SmartTriage we limited ourselves to considering items from the Chief Complaint (CC), History of Present Illness (HPI), Review of Systems (ROS), and Physical Exam (PE) sections, and only considered positive or negative items where the classification probability was >= 90%.
The resulting 25 million encounter records annotated 4,110,052 patients with unique 2,900 chief complaints and 24,472 ICD10 diagnose codes. Tables A.3 and A.4 show the full breakdown of HPI, medical history records, and clinical concepts by gender and age group. 
SmartTriage is currently being piloted at one urgent care facility and there is a plan for an additional, much larger, tele-health pilot to start later this year. If the pilots are successful, Smart-Triage will likely be rolled out widely throughout our health care organization, impacting millions of patients. To enable production deployment, we had to build multiple interfaces with existing enterprise systems. A high-level interaction diagram is presented in Figure B .1.
ETL process for patient history Rather than directly query the EMR, we maintain a local copy of the discrete patient history information for members in our geographic region. We refresh our copy ( 500 million rows) as a nightly extraction process using Informatica.
Encounter start notification For our urgent care pilot, we implemented a real-time HL7 checkin interface from Integration Broker (IB) to SmartTriage that notifies us when a patient is in the clinic. We process 100K check-in messages per day.
Webapp API We implemented a custom REST API to communicate with the enterprise web UI service. For the urgent care pilot, the service notifies the patient about SmartTriage via a text message. When the patient clicks on the link in the text message they are redirected to the the enterprise patient authentication service, and then presented with the patient view of SmartTriage ( Figure B .2).
Inputting data back into the EMR We leverage existing SOAP APIs to send the patient data back into the EMR.
Provider facing interface We implemented a custom UI in the EMR for the clinician to interact with the SmartTriage data, integrate into patient chart, and provide feedback to SmartTriage system (see Figures 1 and 2) .
Daily reporting A daily reporting job is used to extract clinician feedback and collect data for re-training of the machine learning models. 