Literature-Augmented Clinical Outcome Prediction (2022)

Abstract: We present BEEP (Biomedical Evidence-Enhanced Predictions), a novel approach for clinical outcome prediction that retrieves patient-specific medical literature and incorporates it into predictive models. Based on each individual patient’s clinical notes, we train language models (LMs) to find relevant papers and fuse them with information from notes to predict outcomes such as in-hospital mortality. We develop methods to retrieve literature based on noisy, information-dense patient notes, and to augment existing outcome prediction models with retrieved papers in a manner that maximizes predictive accuracy. Our approach boosts predictive performance on three important clinical tasks in comparison to strong recent LM baselines, increasing F1 by up to 5 points and precision@Top-K by a large margin of over 25%.

## introduction predicting the medical outcomes of hospitalized patients holds the promise of enhancing clinical decision making. with the advent of electronic health records (ehrs), more clinical data has become available to train ai models for outcome prediction @xcite . in particular, language models pretrained on biomedical and/or clinical text are demonstrating increasing proficiency when fine-tuned for the task of predicting outcomes such as in-hospital mortality or length of stay @xcite . in this work, we explore a novel approach for improving clinical outcome prediction by dynamically retrieving relevant medical literature for each patient, and incorporating this literature into language models (lms) trained for outcome prediction from clinical notes. this is in contrast to existing outcome prediction work that uses only clinical notes @xcite . recent lm-based approaches van @xcite have designed pretraining schemes over corpora of clinical notes and general biomedical literature. this is in contrast to our work, where we directly incorporate a literature retrieval mechanism into our outcome prediction model, by finding papers relevant to specific patient cases. our approach, named beep (biomedical evidence-enhanced predictions), is broadly inspired by evidence based medicine (ebm)a leading paradigm in modern medical practice which calls for finding the "current best evidence" to support optimal clinical decisions for each individual patient @xcite . our setting presents unique challenges. first, our approach requires retrieving literature based on noisy ehr notes containing multitudes of information (e.g., medical history, ongoing treatments), unlike orthogonal efforts on extracting and summarizing scholarly information related to well-formed questions (e.g., the efficacy of ace inhibitors in adult patients with type-2 diabetes) @xcite @xcite . in addition, as our end task is predicting patient outcomes, another challenge lies in aggregating the retrieved literature in a way that maximizes prediction accuracy. toward these challenges, we make the following key contributions: we hope our work opens new research directions for automatically scanning literature for patientspecific evidence, and combining it with ehr information to boost accuracy of medical predictive models. finally, our work raises the more general prospect of building predictive models that can dynamically learn to retrieve literature for optimizing task accuracy, in medicine and other related areas. ## experiments & results we test our system on the task of predicting clinical outcomes from patient admission notes. predicting outcomes from admission notes can help with early identification of at-risk patients and assist hospitals in resource planning by indicating how long patients may require hospital/icu beds, ventilators etc. @xcite . ## analysis and discussion given beep's improved performance, we further assess the utility of retrieved literature and cases where adding literature is particularly helpful. diversity of retrieved literature. as a preliminary analysis, we evaluate the diversity of the abstracts retrieved for admission notes in our datasets, as a proxy for the degree to which literature is personalized to specific patient cases. for the 100 most frequently retrieved abstracts for each clinical outcome, figures 4a , 4b , and 4c in appendix h show proportions of patient notes for which these abstracts are judged as relevant by our retrievererank pipeline. from these histograms, we see a stark difference for los which is much less diverse than both pmv and mor, indicating that the literature retrieved for length of stay prediction may be less personalized to patient cases than the literature retrieved for other outcomes. we leave to future work exploration of diversifying retrieved papers across patients and examining the effect on outcome prediction performance. 8 qualitative examination of retrieved literature. we qualitatively examine literature retrieved for cases in which our model shows large confidence increases over the baseline to determine its utility in making the right prediction. we study increases in both directions, i.e. cases in which adding literature resulted in a confidence increase in either the correct outcome label (good) or incorrect outcome label (bad). for each clinical outcome, a bio-nlp expert looked at the top 5 cases from each category based on the magnitude of confidence increase (total 10 cases per outcome). for each case, the expert looks at the top 5 abstracts retrieved for the case (total 50 abstracts per outcome) and assigns each abstract to one of 8 categories we define for categorizing degree of relevance and type of evidence provided, including retrievals considered helpful patient ehr retrieved abstract evidence type outcome chief complaint: liver tranplant present illness: ...s/p liver tranplant...dx: esld secondary to alcoholic cirrhosis. medical history: etoh cirrhosis retrospective review of data of 73 consecutive patients with cirrhosis requiring mv...majority of patients, 51/64 (79.7%), dying in the first 48 hours of intubation... patient condition and outcome directly related no pmv chief complaint: aortic dissection present illness: ...72-year-old woman...chest pain...had type a aortic dissection...an intramural hematoma...proceed with surgery... medical history: htn renal failure acute type a aortic dissection presents a formidable challenge...the most important variables associated with inhospital mortality in patients undergoing surgery for this condition...suggests that cpb time, diabetes mellitus and postoperative bleeding are the main determinants of in-hospital death. known outcome indicators not present in patient no mor chief complaint: dyspnea, fever present illness: 58f w/ hcv cirrhosis...requiring bipap, ultimately urgent intubation... extubated ... short of breath... medical history: hcv cirrhosis ...study identifies specific predictors of increased mortality and resource utilization in cirrhotic patients...increased los in the micu was associated with mechanical ventilation... and unhelpful. for example, see as seen in figure 3 presents the distribution of helpful and unhelpful categories for both kinds of cases for all outcomes. we can see that for correct outcome cases from both pmv and mortality, retrieved literature is more frequently assigned to one of the helpful categories, while for incorrect outcome cases, retrieved literature is more frequently assigned to one of the unhelpful categories. for los, unhelpful categories dominate both types of cases, especially prevalent in incorrect outcomes. ## conclusion in this paper, we introduced beep, a system that automatically retrieves patient-specific literature based on intensive care (icu) ehr notes and uses the literature to enhance clinical outcome prediction. on three challenging tasks, we obtain substantial improvements over strong recent baselines, seeing dramatic gains in top-10% precision for mortality prediction with a boost of over 25%. our hope is that this work will open new research directions into bridging the gap between ai-based clinical models and the evidence based medicine (ebm) paradigm in which medical decisions are based on explicit evidence from the literature. an interesting direction is to incorporate evidence identification and inference @xcite directly into our retrieval and predictive models. another important question to explore relates to the implications our approach has on increasing the interpretability of clinical ai models.