Towards Generalizable Methods for Automating Risk Score Calculation (2022)

Abstract: Clinical risk scores enable clinicians to tabulate a set of patient data into simple scores to stratify patients into risk categories. Although risk scores are widely used to inform decision-making at the point-of-care, collecting the information necessary to calculate such scores requires considerable time and effort. Previous studies have focused on specific risk scores and involved manual curation of relevant terms or codes and heuristics for each data element of a risk score. To support more generalizable methods for risk score calculation, we annotate 100 patients in MIMIC-III with elements of CHA2DS2-VASc and PERC scores, and explore using question answering (QA) and off-the-shelf tools. We show that QA models can achieve comparable or better performance for certain risk score elements as compared to heuristic-based methods, and demonstrate the potential for more scalable risk score automation without the need for expert-curated heuristics. Our annotated dataset will be released to the community to encourage efforts in generalizable methods for automating risk scores.

## introduction clinical risk scores are standardized metrics to estimate the risk of a particular future outcome based on available clinical parameters and are commonly used at the point-of-care to inform decision-making around diagnosis and treatment @xcite . an example of this is the cha 2 ds 2 -vasc score @xcite , which uses 7 patient data elements to estimate the risk of stroke in patients with non-valvular atrial fibrillation and thus guide strategies around stroke prevention. it has successfully demonstrated clinical impact and is referenced in the practice guidelines for management of atrial fibrillation released by the american heart association, american college of cardiology, and heart rhythm society in 2014 @xcite . in general, data elements that contribute to a risk score may include information about the patient's age, gender, medical history, presenting symptoms, medication use, etc. while risk scores are generally designed for use at the point-of-care, calculating them can require considerable time and effort, as each data element must be manually gathered, often from multiple locations within the electronic health record (ehr). a previous study investigating the feasibility of automating clinical score calculation identified 534 unique patient data elements from 168 externally validated clinical scores, with each score requiring anywhere from 3 to 31 elements @xcite . automating extraction of clinical data elements necessary to calculate risk scores could save clinicians time and help them more effectively leverage risk scores to improve care at the bedside @xcite . cha₂ds₂-vasc diabetes mellitus? chf history? htn history? tia history? vascular disease? medcat structured data row_id subject_id admittime icd9 0 00001 2021-01-28 20294 1 00002 2022-04-07 25000 2 00003 2022-01-21 90832 3 00004 2022-02-10 62849 given calcium, insulin 12:40 am insulin -regular patient has h/o diabetes for physician review @xmath0: jane doe, md patient: patient a sex: female lorem ipsum dolor sit amet, alii torquatos id per, eam dicat reprehendunt ut. suas diam ad nec. an cum solum paulo diceret, eu vide splendide rationibus vis, dicat the cha 2 ds 2 -vasc score over unstructured ehr data. while these efforts found strong agreement with expert human evaluators, heuristic-based approaches are often rigid and struggle to generalize to other problems. thus, we propose using offthe-shelf tools and pretrained language models to extract evidence from both structured and unstructured ehr data, without the need for manuallycurated rules. in this study, we explore two commonly used risk scores -the cha 2 ds 2 -vasc score for atrial fibrillation stroke risk @xcite and the pulmonary embolism rule-out criteria (perc) rule @xcite -to demonstrate our approach. we use a transformer-based model trained on emrqa @xcite and an off-theshelf biomedical ontology linker paired with a sql query component to extract evidence from unstructured and structured ehr data, respectively, for each element of the two risk scores (figure 1 ). the main contributions of this work are: to evaluate our models, we randomly sample 100 patients from the medical information mart for intensive care iii (mimic-iii) dataset (johnson et al., 2016) to annotate with elements of cha 2 ds 2 -vasc and perc. cha 2 ds 2 -vasc uses 7 patient data elements to estimate the risk of stroke in patients with non-valvular atrial fibrillation: congestive heart failure (chf), hypertension, age, diabetes mellitus, stroke/transient ischemic attack (tia)/thromboembolism (te), vascular disease (prior myocardial infarction, peripheral artery disease, or aortic plaque), and sex. perc uses 8 elements to evaluate the risk of pe in low-risk patients: age, heart rate, oxygen saturation, unilateral leg swelling, hemoptysis, recent surgery or trauma, prior pe or deep venous thrombosis (dvt), and hormone use. we frame our scenario as a new patient being seen in the emergency department (ed) requiring calculation of cha 2 ds 2 -vasc or perc because of suspected atrial fibrillation or pe, respectively, and the data in mimic-iii is the available past medical history for this patient. therefore, we limited our dataset to non-expired patients at least 18 years of age at time of last discharge with at least one discharge summary. since perc only rules out pe when none of the criteria are met, one of which is @xmath1, we further adjust our sampling such that at least half of the patients selected are under 50 years of age at time of last discharge to ensure non-trivial calculation of perc. the dataset was annotated by two independent annotators, with a 20% overlap for inter-annotator agreement (κ = 0.800), and then reviewed by a physician. annotators reviewed the entire ehr data provided in mimic-iii, including both structured and unstructured sources, and annotated evidence relevant to each risk score element. evidence in structured data include coded diagnoses, procedures, and past medical history. evidence in unstructured data consist of text snippets from discharge summaries, admission notes, progress notes, and their addenda. patients in our subset had an average of 44 notes with average length of 289 tokens. since we frame our scenario as a new patient being seen in the ed, vital signs (e.g., heart rate, oxygen saturation) as recorded in their history (i.e., mimic-iii in our scenario) would not be relevant and are therefore excluded from annotation. for other elements in perc that may also be timesensitive, since the exact time frame is not always apparent from the given documentation, for the purposes of this study, we annotate all instances of unilateral leg swelling, hemoptysis, surgery and trauma as evidence for their respective elements regardless of when they occurred. in addition to the evidence, annotators also provided an overall answer for each risk score element: "yes", "no", "unclear" (evidence present but conflicting or inconclusive), or "no data". a sample of the annotated dataset is presented in ## task setup to extract information relevant to the specified risk score, we query the system with risk score elements expressed as short natural language phrases containing the entities (e.g., "hypertension"). elements containing multiple concepts are split into multiple phrases, each containing a single concept. for the purposes of evaluation, "yes" and "unclear" in the ground truth are considered to be equivalent because both provide some positive evidence, while "no" and "no data" are considered to be equivalent because in practice, lack of data would be presumed to be negative. for unstructured data, a system is tasked with predicting the presence or absence of the given risk score element. the system must also provide the sentence it selected to make its decision. predictions considered true positives when compared to the ground truth are further reviewed by a physician to ensure that the sentence used for prediction can reasonably be used to determine if the patient has the given condition; if the sentence used for prediction cannot be used to logically determine whether the patient has the given condition, the prediction is marked as a false positive. for structured data, the model is tasked with retrieving a yes/no answer along with the relevant billing code (when present) for each risk score element. we evaluate the system by matching the retrieved yes/no answer with the ground truth, and calculating the precision, recall, and f1-score. ## discussion we make a few key observations. we find that the structured data model is able to achieve extremely high performance in a number categories, but unable to find any relevant information for the rest. we hypothesize that this is due to chronic conditions (e.g., chf, hypertension) being more consistently recorded in the structured data, while acute events (e.g., pe/dvt, stroke/tia/te) are coded only in the limited time frame when such conditions are being actively managed. also, structured data, in the form of billing codes, would not be expected to capture symptoms without a formal diagnosis (e.g., unilateral leg swelling). we additionally find that the qa model on unstructured data alone is able to improve on the results of @xcite on a number of categories, without the need for expert-crafted heuristics. however, we find that the qa model struggles due to a lack of clinical knowledge and ability to distinguish hypothetical mentions versus true affirmations of the given condition. with respect to vascular disease, an error analysis of the qa-based model showed that 69% of the false positives were due to a lack of clinical understanding, as the model considered a much broader definition of vascular disease than the one specified in the cha 2 ds 2 -vasc score. similarly, with respect to stroke/tia/te, we find that 93% of the false positives can be attributed to imprecise understanding of medical terminology and the model's inability to use contextual clues to differentiate between stroke and other conditions. we additionally see extremely low precision for pe/dvt. this can largely be attributed to faulty negation detection, as medcat often fails to distinguish between affirming and hypothetical/negated mentions in over 70% of the false positives. one issue we found when implementing @xcite 's approach is that it is nontrivial to determine which cuis to select, specifically for general categories like surgery and trauma. using all umls descendants of surgery and trauma results in 3,413,446 unique cuis, which will clearly result in an enormous number of false positive results when selecting sentences, as seen in ## conclusion we explore risk score automation using qa and off-the-shelf ontology entity linkers without the need for expert-curated rules, and demonstrate its potential for easy adaptation to unexplored risk scores. we find that qa models can achieve comparable or better performance for certain risk score elements as compared to heuristic-based methods, and demonstrate the potential for more scalable risk score automation without the need for expertcurated heuristics. our annotated dataset will be released to the community to encourage efforts in generalizable methods for automating risk scores.