A sepsis early warning system is associated with improved patient outcomes
Authors: Jason N. Kennedy, Kristina E. Rudd
Journal: Cell Reports Medicine (2022)

In a real-world implementation of a machine-learning (ML)-based sepsis early warning system (EWS), Adams et al.1Adams R. Henry K.E. Sridharan A. Soleimani H. Zhan A. Rawat N. Johnson L. Hager D.N. Cosgrove S.E. Markowski A. et al.Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis.Nat. Med. 2022; 28: 1455-1460https://doi.org/10.1038/s41591-022-01894-0Crossref PubMed Scopus (5) Google Scholar, 2Marshall J.C. Why have clinical trials in sepsis failed?.Trends Mol. Med. 2014; 20: 195-203https://doi.org/10.1016/j.molmed.2014.01.007Abstract Full Text Full Text PDF PubMed Scopus (395) Google Scholar found that timely provider response to an alert was associated with improved mortality, highlighting the potential utility of these systems in patient care. In a real-world implementation of a machine-learning (ML)-based sepsis early warning system (EWS), Adams et al.1Adams R. Henry K.E. Sridharan A. Soleimani H. Zhan A. Rawat N. Johnson L. Hager D.N. Cosgrove S.E. Markowski A. et al.Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis.Nat. Med. 2022; 28: 1455-1460https://doi.org/10.1038/s41591-022-01894-0Crossref PubMed Scopus (5) Google Scholar, 2Marshall J.C. Why have clinical trials in sepsis failed?.Trends Mol. Med. 2014; 20: 195-203https://doi.org/10.1016/j.molmed.2014.01.007Abstract Full Text Full Text PDF PubMed Scopus (395) Google Scholar found that timely provider response to an alert was associated with improved mortality, highlighting the potential utility of these systems in patient care. Main textDespite being a leading cause of hospitalization and death worldwide, there are few available interventions to improve patient outcomes in sepsis.2Marshall J.C. Why have clinical trials in sepsis failed?.Trends Mol. Med. 2014; 20: 195-203https://doi.org/10.1016/j.molmed.2014.01.007Abstract Full Text Full Text PDF PubMed Scopus (395) Google Scholar Multiple studies have shown that earlier sepsis recognition and antibiotic administration is associated with improved survival,3Seymour C.W. Gesten F. Prescott H.C. Friedrich M.E. Iwashyna T.J. Phillips G.S. Lemeshow S. Osborn T. Terry K.M. Levy M.M. Time to treatment and mortality during Mandated emergency care for sepsis.N. Engl. J. Med. 2017; 376: 2235-2244https://doi.org/10.1056/NEJMoa1703058Crossref PubMed Scopus (947) Google Scholar,4Liu V.X. Fielding-Singh V. Greene J.D. Baker J.M. Iwashyna T.J. Bhattacharya J. Escobar G.J. The timing of early antibiotics and hospital mortality in sepsis.Am. J. Respir. Crit. Care Med. 2017; 196: 856-863https://doi.org/10.1164/rccm.201609-1848OCCrossref PubMed Scopus (346) Google Scholar but development of a standard best practice to achieve this has been elusive. One small randomized trial found that use of a sepsis early warning system (EWS) was associated with reduced mortality and length of stay (LOS) among intensive care unit (ICU) patients.5Shimabukuro D.W. Barton C.W. Feldman M.D. Mataraso S.J. Das R. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial.BMJ Open Respir. Res. 2017; 4: e000234https://doi.org/10.1136/bmjresp-2017-000234Crossref PubMed Scopus (142) Google Scholar For a syndrome where more than 100 clinical trials have failed,2Marshall J.C. Why have clinical trials in sepsis failed?.Trends Mol. Med. 2014; 20: 195-203https://doi.org/10.1016/j.molmed.2014.01.007Abstract Full Text Full Text PDF PubMed Scopus (395) Google Scholar EWS may thus be one tool to improve outcomes.EWSs are part of a broader family of clinical decision support tools, whose use for sepsis dates back to the 1970s.6Shortliffe E.H. Axline S.G. Buchanan B.G. Merigan T.C. Cohen S.N. An Artificial Intelligence program to advise physicians regarding antimicrobial therapy.Comput. Biomed. Res. 1973; 6: 544-560https://doi.org/10.1016/0010-4809(73)90029-3Crossref PubMed Scopus (124) Google Scholar Underlying models are most commonly developed in retrospective or administrative data, can be used for risk modeling or detection of clinical deterioration, and can vary from regression-based approaches to complex neural networks.7Sutton R.T. Pincock D. Baumgart D.C. Sadowski D.C. Fedorak R.N. Kroeker K.I. An overview of clinical decision support systems: benefits, risks, and strategies for success.npj Digital Med. 2020; 3: 17https://doi.org/10.1038/s41746-020-0221-yCrossref PubMed Scopus (331) Google Scholar,8Reddy S. Fox J. Purohit M.P. Artificial intelligence-enabled healthcare delivery.J. R. Soc. Med. 2019; 112: 22-28https://doi.org/10.1177/0141076818815510Crossref PubMed Scopus (112) Google Scholar EWSs vary in intended clinical environment, with systems designed for emergency departments (EDs), ICUs, and surgical units, among others. While many models have been retrospectively described, few have been evaluated prospectively in a clinical setting.In a recent issue of Nature Medicine, Adams and colleagues report the results of a prospective, multi-center, two-arm cohort study evaluating the association between prompt provider confirmation of an EHR-embedded sepsis alert and patient outcomes.1Adams R. Henry K.E. Sridharan A. Soleimani H. Zhan A. Rawat N. Johnson L. Hager D.N. Cosgrove S.E. Markowski A. et al.Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis.Nat. Med. 2022; 28: 1455-1460https://doi.org/10.1038/s41591-022-01894-0Crossref PubMed Scopus (5) Google Scholar The alert system, targeted real-time early warning system (TREWS), is an ML-based system deployed within two academic and three community hospitals in the US. TREWS continuously monitors patient vitals, laboratory data, medication orders, and clinical documentation to generate a real-time sepsis risk score.9Henry K.E. Kornfield R. Sridharan A. Linton R.C. Groh C. Wang T. Wu A. Mutlu B. Saria S. Human-machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system.npj Digital Med. 2022; 5: 97https://doi.org/10.1038/s41746-022-00597-7Crossref PubMed Scopus (4) Google Scholar When an alert occurs, providers can either dismiss it or proceed to a dedicated page with further information. There, the provider enters an evaluation either confirming suspected infection or indicating they do not believe new or worsening infection is present.From TREWS deployment in 2018 through September 2020, there were 590,736 adult inpatient encounters across the five hospitals. Of these, 6,877 met inclusion criteria: they triggered TREWS alert no more than 1 h prior to admission or ED triage, met retrospective sepsis criteria, received antibiotics not prior to but within 24 h after the alert, and were not admitted directly to the ICU. Among these 6,877 encounters, 4,220 (61%) had their TREWS alert evaluated and confirmed as suspected sepsis within 3 h (study arm) and 2,657 (39%) did not (comparison arm).The primary outcome was adjusted all-cause hospital mortality. Secondary outcomes included change in SOFA score in the 72 h after alert and post-alert hospital LOS among survivors. These outcomes were adjusted for admitting hospital and patient demographics, comorbidities, and severity of illness. No adjustment was made for provider-level features. Alert confirmation within 3 h was associated with lower mortality, with 14.6% unadjusted mortality in the study arm and 19.2% in the comparison arm; adjusted mortality risk difference was −3.3% (CI −5.1, −1.7%, p < 0.001). There were significant differences between groups in both secondary outcomes: adjusted change in SOFA score in the 72 h after alert was −0.3 points (CI −.4, −0.1, p = 0.001) and adjusted median post-alert LOS among survivors was −11.6 h (CI, −18.1, −5.0, p = 0.001).This is the first study of this scale to prospectively demonstrate the potential for ML-based alert systems to identify patients with sepsis early and meaningfully improve outcomes. Strengths of the study include its size, multi-center design, and low risk of surveillance bias through use of a natural contemporaneous control group. Additionally, use of "human-machine teaming" to integrate clinician impression made TREWS feel like more of a tool for care, rather than simply an alert.9Henry K.E. Kornfield R. Sridharan A. Linton R.C. Groh C. Wang T. Wu A. Mutlu B. Saria S. Human-machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system.npj Digital Med. 2022; 5: 97https://doi.org/10.1038/s41746-022-00597-7Crossref PubMed Scopus (4) Google ScholarImportant questions remain. While the EWS may indeed have led to earlier treatment, and thus to improved patient outcomes, it is possible that prompt response to the alert was simply indicative of a more attentive or better resourced care team in which we would already expect improved patient outcomes. While the authors appropriately adjusted for patient-level variability, this provider variability is more difficult to account for in observational data. A future study in which implementation is randomized by site or provider, such as a step-wedge design, may be helpful for reducing this potential confounding. In addition, the alert system was triggered 42,089 times, but sepsis criteria were met in only 13,680 encounters.1Adams R. Henry K.E. Sridharan A. Soleimani H. Zhan A. Rawat N. Johnson L. Hager D.N. Cosgrove S.E. Markowski A. et al.Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis.Nat. Med. 2022; 28: 1455-1460https://doi.org/10.1038/s41591-022-01894-0Crossref PubMed Scopus (5) Google Scholar While companion papers sought to understand factors to increase alert usage, further work in both model refinement and implementation science is needed to ensure alert usage is maintained over time and not impacted by alert fatigue.9Henry K.E. Kornfield R. Sridharan A. Linton R.C. Groh C. Wang T. Wu A. Mutlu B. Saria S. Human-machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system.npj Digital Med. 2022; 5: 97https://doi.org/10.1038/s41746-022-00597-7Crossref PubMed Scopus (4) Google Scholar,10Henry K.E. Adams R. Parent C. Soleimani H. Sridharan A. Johnson L. Hager D.N. Cosgrove S.E. Markowski A. Klein E.Y. et al.Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing.Nat. Med. 2022; 28: 1447-1454https://doi.org/10.1038/s41591-022-01895-zCrossref PubMed Scopus (5) Google ScholarAs use of EWS expands, there will be challenges in deployment into EHR systems that vary substantially between hospitals. The authors successfully implemented TREWS across 5 sites with a shared EHR, but there is not yet a straightforward way to scale implementation across dissimilar systems. Continued advancement of Health Level 7 (HL7) and Fast Healthcare Interoperability Resource (FHIR) standards offers exciting potential for the future development of Substitutable Medical Applications and Reusable Technologies (SMART).In summary, real-time EWS such as the one created by Adams and colleagues have potential to improve patient outcomes by prompting earlier and more appropriate clinical care. Future work is needed to evaluate these systems in large-scale randomized trials, to better understand the human factors driving adoption and action at the provider level, and to develop technological frameworks to facilitate broader clinical implementation. Main textDespite being a leading cause of hospitalization and death worldwide, there are few available interventions to improve patient outcomes in sepsis.2Marshall J.C. Why have clinical trials in sepsis failed?.Trends Mol. Med. 2014; 20: 195-203https://doi.org/10.1016/j.molmed.2014.01.007Abstract Full Text Full Text PDF PubMed Scopus (395) Google Scholar Multiple studies have shown that earlier sepsis recognition and antibiotic administration is associated with improved survival,3Seymour C.W. Gesten F. Prescott H.C. Friedrich M.E. Iwashyna T.J. Phillips G.S. Lemeshow S. Osborn T. Terry K.M. Levy M.M. Time to treatment and mortality during Mandated emergency care for sepsis.N. Engl. J. Med. 2017; 376: 2235-2244https://doi.org/10.1056/NEJMoa1703058Crossref PubMed Scopus (947) Google Scholar,4Liu V.X. Fielding-Singh V. Greene J.D. Baker J.M. Iwashyna T.J. Bhattacharya J. Escobar G.J. The timing of early antibiotics and hospital mortality in sepsis.Am. J. Respir. Crit. Care Med. 2017; 196: 856-863https://doi.org/10.1164/rccm.201609-1848OCCrossref PubMed Scopus (346) Google Scholar but development of a standard best practice to achieve this has been elusive. One small randomized trial found that use of a sepsis early warning system (EWS) was associated with reduced mortality and length of stay (LOS) among intensive care unit (ICU) patients.5Shimabukuro D.W. Barton C.W. Feldman M.D. Mataraso S.J. Das R. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial.BMJ Open Respir. Res. 2017; 4: e000234https://doi.org/10.1136/bmjresp-2017-000234Crossref PubMed Scopus (142) Google Scholar For a syndrome where more than 100 clinical trials have failed,2Marshall J.C. Why have clinical trials in sepsis failed?.Trends Mol. Med. 2014; 20: 195-203https://doi.org/10.1016/j.molmed.2014.01.007Abstract Full Text Full Text PDF PubMed Scopus (395) Google Scholar EWS may thus be one tool to improve outcomes.EWSs are part of a broader family of clinical decision support tools, whose use for sepsis dates back to the 1970s.6Shortliffe E.H. Axline S.G. Buchanan B.G. Merigan T.C. Cohen S.N. An Artificial Intelligence program to advise physicians regarding antimicrobial therapy.Comput. Biomed. Res. 1973; 6: 544-560https://doi.org/10.1016/0010-4809(73)90029-3Crossref PubMed Scopus (124) Google Scholar Underlying models are most commonly developed in retrospective or administrative data, can be used for risk modeling or detection of clinical deterioration, and can vary from regression-based approaches to complex neural networks.7Sutton R.T. Pincock D. Baumgart D.C. Sadowski D.C. Fedorak R.N. Kroeker K.I. An overview of clinical decision support systems: benefits, risks, and strategies for success.npj Digital Med. 2020; 3: 17https://doi.org/10.1038/s41746-020-0221-yCrossref PubMed Scopus (331) Google Scholar,8Reddy S. Fox J. Purohit M.P. Artificial intelligence-enabled healthcare delivery.J. R. Soc. Med. 2019; 112: 22-28https://doi.org/10.1177/0141076818815510Crossref PubMed Scopus (112) Google Scholar EWSs vary in intended clinical environment, with systems designed for emergency departments (EDs), ICUs, and surgical units, among others. While many models have been retrospectively described, few have been evaluated prospectively in a clinical setting.In a recent issue of Nature Medicine, Adams and colleagues report the results of a prospective, multi-center, two-arm cohort study evaluating the association between prompt provider confirmation of an EHR-embedded sepsis alert and patient outcomes.1Adams R. Henry K.E. Sridharan A. Soleimani H. Zhan A. Rawat N. Johnson L. Hager D.N. Cosgrove S.E. Markowski A. et al.Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis.Nat. Med. 2022; 28: 1455-1460https://doi.org/10.1038/s41591-022-01894-0Crossref PubMed Scopus (5) Google Scholar The alert system, targeted real-time early warning system (TREWS), is an ML-based system deployed within two academic and three community hospitals in the US. TREWS continuously monitors patient vitals, laboratory data, medication orders, and clinical documentation to generate a real-time sepsis risk score.9Henry K.E. Kornfield R. Sridharan A. Linton R.C. Groh C. Wang T. Wu A. Mutlu B. Saria S. Human-machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system.npj Digital Med. 2022; 5: 97https://doi.org/10.1038/s41746-022-00597-7Crossref PubMed Scopus (4) Google Scholar When an alert occurs, providers can either dismiss it or proceed to a dedicated page with further information. There, the provider enters an evaluation either confirming suspected infection or indicating they do not believe new or worsening infection is present.From TREWS deployment in 2018 through September 2020, there were 590,736 adult inpatient encounters across the five hospitals. Of these, 6,877 met inclusion criteria: they triggered TREWS alert no more than 1 h prior to admission or ED triage, met retrospective sepsis criteria, received antibiotics not prior to but within 24 h after the alert, and were not admitted directly to the ICU. Among these 6,877 encounters, 4,220 (61%) had their TREWS alert evaluated and confirmed as suspected sepsis within 3 h (study arm) and 2,657 (39%) did not (comparison arm).The primary outcome was adjusted all-cause hospital mortality. Secondary outcomes included change in SOFA score in the 72 h after alert and post-alert hospital LOS among survivors. These outcomes were adjusted for admitting hospital and patient demographics, comorbidities, and severity of illness. No adjustment was made for provider-level features. Alert confirmation within 3 h was associated with lower mortality, with 14.6% unadjusted mortality in the study arm and 19.2% in the comparison arm; adjusted mortality risk difference was −3.3% (CI −5.1, −1.7%, p < 0.001). There were significant differences between groups in both secondary outcomes: adjusted change in SOFA score in the 72 h after alert was −0.3 points (CI −.4, −0.1, p = 0.001) and adjusted median post-alert LOS among survivors was −11.6 h (CI, −18.1, −5.0, p = 0.001).This is the first study of this scale to prospectively demonstrate the potential for ML-based alert systems to identify patients with sepsis early and meaningfully improve outcomes. Strengths of the study include its size, multi-center design, and low risk of surveillance bias through use of a natural contemporaneous control group. Additionally, use of "human-machine teaming" to integrate clinician impression made TREWS feel like more of a tool for care, rather than simply an alert.9Henry K.E. Kornfield R. Sridharan A. Linton R.C. Groh C. Wang T. Wu A. Mutlu B. Saria S. Human-machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system.npj Digital Med. 2022; 5: 97https://doi.org/10.1038/s41746-022-00597-7Crossref PubMed Scopus (4) Google ScholarImportant questions remain. While the EWS may indeed have led to earlier treatment, and thus to improved patient outcomes, it is possible that prompt response to the alert was simply indicative of a more attentive or better resourced care team in which we would already expect improved patient outcomes. While the authors appropriately adjusted for patient-level variability, this provider variability is more difficult to account for in observational data. A future study in which implementation is randomized by site or provider, such as a step-wedge design, may be helpful for reducing this potential confounding. In addition, the alert system was triggered 42,089 times, but sepsis criteria were met in only 13,680 encounters.1Adams R. Henry K.E. Sridharan A. Soleimani H. Zhan A. Rawat N. Johnson L. Hager D.N. Cosgrove S.E. Markowski A. et al.Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis.Nat. Med. 2022; 28: 1455-1460https://doi.org/10.1038/s41591-022-01894-0Crossref PubMed Scopus (5) Google Scholar While companion papers sought to understand factors to increase alert usage, further work in both model refinement and implementation science is needed to ensure alert usage is maintained over time and not impacted by alert fatigue.9Henry K.E. Kornfield R. Sridharan A. Linton R.C. Groh C. Wang T. Wu A. Mutlu B. Saria S. Human-machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system.npj Digital Med. 2022; 5: 97https://doi.org/10.1038/s41746-022-00597-7Crossref PubMed Scopus (4) Google Scholar,10Henry K.E. Adams R. Parent C. Soleimani H. Sridharan A. Johnson L. Hager D.N. Cosgrove S.E. Markowski A. Klein E.Y. et al.Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing.Nat. Med. 2022; 28: 1447-1454https://doi.org/10.1038/s41591-022-01895-zCrossref PubMed Scopus (5) Google ScholarAs use of EWS expands, there will be challenges in deployment into EHR systems that vary substantially between hospitals. The authors successfully implemented TREWS across 5 sites with a shared EHR, but there is not yet a straightforward way to scale implementation across dissimilar systems. Continued advancement of Health Level 7 (HL7) and Fast Healthcare Interoperability Resource (FHIR) standards offers exciting potential for the future development of Substitutable Medical Applications and Reusable Technologies (SMART).In summary, real-time EWS such as the one created by Adams and colleagues have potential to improve patient outcomes by prompting earlier and more appropriate clinical care. Future work is needed to evaluate these systems in large-scale randomized trials, to better understand the human factors driving adoption and action at the provider level, and to develop technological frameworks to facilitate broader clinical implementation. Despite being a leading cause of hospitalization and death worldwide, there are few available interventions to improve patient outcomes in sepsis.2Marshall J.C. Why have clinical trials in sepsis failed?.Trends Mol. Med. 2014; 20: 195-203https://doi.org/10.1016/j.molmed.2014.01.007Abstract Full Text Full Text PDF PubMed Scopus (395) Google Scholar Multiple studies have shown that earlier sepsis recognition and antibiotic administration is associated with improved survival,3Seymour C.W. Gesten F. Prescott H.C. Friedrich M.E. Iwashyna T.J. Phillips G.S. Lemeshow S. Osborn T. Terry K.M. Levy M.M. Time to treatment and mortality during Mandated emergency care for sepsis.N. Engl. J. Med. 2017; 376: 2235-2244https://doi.org/10.1056/NEJMoa1703058Crossref PubMed Scopus (947) Google Scholar,4Liu V.X. Fielding-Singh V. Greene J.D. Baker J.M. Iwashyna T.J. Bhattacharya J. Escobar G.J. The timing of early antibiotics and hospital mortality in sepsis.Am. J. Respir. Crit. Care Med. 2017; 196: 856-863https://doi.org/10.1164/rccm.201609-1848OCCrossref PubMed Scopus (346) Google Scholar but development of a standard best practice to achieve this has been elusive. One small randomized trial found that use of a sepsis early warning system (EWS) was associated with reduced mortality and length of stay (LOS) among intensive care unit (ICU) patients.5Shimabukuro D.W. Barton C.W. Feldman M.D. Mataraso S.J. Das R. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial.BMJ Open Respir. Res. 2017; 4: e000234https://doi.org/10.1136/bmjresp-2017-000234Crossref PubMed Scopus (142) Google Scholar For a syndrome where more than 100 clinical trials have failed,2Marshall J.C. Why have clinical trials in sepsis failed?.Trends Mol. Med. 2014; 20: 195-203https://doi.org/10.1016/j.molmed.2014.01.007Abstract Full Text Full Text PDF PubMed Scopus (395) Google Scholar EWS may thus be one tool to improve outcomes. EWSs are part of a broader family of clinical decision support tools, whose use for sepsis dates back to the 1970s.6Shortliffe E.H. Axline S.G. Buchanan B.G. Merigan T.C. Cohen S.N. An Artificial Intelligence program to advise physicians regarding antimicrobial therapy.Comput. Biomed. Res. 1973; 6: 544-560https://doi.org/10.1016/0010-4809(73)90029-3Crossref PubMed Scopus (124) Google Scholar Underlying models are most commonly developed in retrospective or administrative data, can be used for risk modeling or detection of clinical deterioration, and can vary from regression-based approaches to complex neural networks.7Sutton R.T. Pincock D. Baumgart D.C. Sadowski D.C. Fedorak R.N. Kroeker K.I. An overview of clinical decision support systems: benefits, risks, and strategies for success.npj Digital Med. 2020; 3: 17https://doi.org/10.1038/s41746-020-0221-yCrossref PubMed Scopus (331) Google Scholar,8Reddy S. Fox J. Purohit M.P. Artificial intelligence-enabled healthcare delivery.J. R. Soc. Med. 2019; 112: 22-28https://doi.org/10.1177/0141076818815510Crossref PubMed Scopus (112) Google Scholar EWSs vary in intended clinical environment, with systems designed for emergency departments (EDs), ICUs, and surgical units, among others. While many models have been retrospectively described, few have been evaluated prospectively in a clinical setting. In a recent issue of Nature Medicine, Adams and colleagues report the results of a prospective, multi-center, two-arm cohort study evaluating the association between prompt provider confirmation of an EHR-embedded sepsis alert and patient outcomes.1Adams R. Henry K.E. Sridharan A. Soleimani H. Zhan A. Rawat N. Johnson L. Hager D.N. Cosgrove S.E. Markowski A. et al.Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis.Nat. Med. 2022; 28: 1455-1460https://doi.org/10.1038/s41591-022-01894-0Crossref PubMed Scopus (5) Google Scholar The alert system, targeted real-time early warning system (TREWS), is an ML-based system deployed within two academic and three community hospitals in the US. TREWS continuously monitors patient vitals, laboratory data, medication orders, and clinical documentation to generate a real-time sepsis risk score.9Henry K.E. Kornfield R. Sridharan A. Linton R.C. Groh C. Wang T. Wu A. Mutlu B. Saria S. Human-machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system.npj Digital Med. 2022; 5: 97https://doi.org/10.1038/s41746-022-00597-7Crossref PubMed Scopus (4) Google Scholar When an alert occurs, providers can either dismiss it or proceed to a dedicated page with further information. There, the provider enters an evaluation either confirming suspected infection or indicating they do not believe new or worsening infection is present. From TREWS deployment in 2018 through September 2020, there were 590,736 adult inpatient encounters across the five hospitals. Of these, 6,877 met inclusion criteria: they triggered TREWS alert no more than 1 h prior to admission or ED triage, met retrospective sepsis criteria, received antibiotics not prior to but within 24 h after the alert, and were not admitted directly to the ICU. Among these 6,877 encounters, 4,220 (61%) had their TREWS alert evaluated and confirmed as suspected sepsis within 3 h (study arm) and 2,657 (39%) did not (comparison arm). The primary outcome was adjusted all-cause hospital mortality. Secondary outcomes included change in SOFA score in the 72 h after alert and post-alert hospital LOS among survivors. These outcomes were adjusted for admitting hospital and patient demographics, comorbidities, and severity of illness. No adjustment was made for provider-level features. Alert confirmation within 3 h was associated with lower mortality, with 14.6% unadjusted mortality in the study arm and 19.2% in the comparison arm; adjusted mortality risk difference was −3.3% (CI −5.1, −1.7%, p < 0.001). There were significant differences between groups in both secondary outcomes: adjusted change in SOFA score in the 72 h after alert was −0.3 points (CI −.4, −0.1, p = 0.001) and adjusted median post-alert LOS among survivors was −11.6 h (CI, −18.1, −5.0, p = 0.001). This is the first study of this scale to prospectively demonstrate the potential for ML-based alert systems to identify patients with sepsis early and meaningfully improve outcomes. Strengths of the study include its size, multi-center design, and low risk of surveillance bias through use of a natural contemporaneous control group. Additionally, use of "human-machine teaming" to integrate clinician impression made TREWS feel like more of a tool for care, rather than simply an alert.9Henry K.E. Kornfield R. Sridharan A. Linton R.C. Groh C. Wang T. Wu A. Mutlu B. Saria S. Human-machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system.npj Digital Med. 2022; 5: 97https://doi.org/10.1038/s41746-022-00597-7Crossref PubMed Scopus (4) Google Scholar Important questions remain. While the EWS may indeed have led to earlier treatment, and thus to improved patient outcomes, it is possible that prompt response to the alert was simply indicative of a more attentive or better resourced care team in which we would already expect improved patient outcomes. While the authors appropriately adjusted for patient-level variability, this provider variability is more difficult to account for in observational data. A future study in which implementation is randomized by site or provider, such as a step-wedge design, may be helpful for reducing this potential confounding. In addition, the alert system was triggered 42,089 times, but sepsis criteria were met in only 13,680 encounters.1Adams R. Henry K.E. Sridharan A. Soleimani H. Zhan A. Rawat N. Johnson L. Hager D.N. Cosgrove S.E. Markowski A. et al.Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis.Nat. Med. 2022; 28: 1455-1460https://doi.org/10.1038/s41591-022-01894-0Crossref PubMed Scopus (5) Google Scholar While companion papers sought to understand factors to increase alert usage, further work in both model refinement and implementation science is needed to ensure alert usage is maintained over time and not impacted by alert fatigue.9Henry K.E. Kornfield R. Sridharan A. Linton R.C. Groh C. Wang T. Wu A. Mutlu B. Saria S. Human-machine teaming is key to AI adoption: clinicians' experiences with a deployed machine learning system.npj Digital Med. 2022; 5: 97https://doi.org/10.1038/s41746-022-00597-7Crossref PubMed Scopus (4) Google Scholar,10Henry K.E. Adams R. Parent C. Soleimani H. Sridharan A. Johnson L. Hager D.N. Cosgrove S.E. Markowski A. Klein E.Y. et al.Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing.Nat. Med. 2022; 28: 1447-1454https://doi.org/10.1038/s41591-022-01895-zCrossref PubMed Scopus (5) Google Scholar As use of EWS expands, there will be challenges in deployment into EHR systems that vary substantially between hospitals. The authors successfully implemented TREWS across 5 sites with a shared EHR, but there is not yet a straightforward way to scale implementation across dissimilar systems. Continued advancement of Health Level 7 (HL7) and Fast Healthcare Interoperability Resource (FHIR) standards offers exciting potential for the future development of Substitutable Medical Applications and Reusable Technologies (SMART). In summary, real-time EWS such as the one created by Adams and colleagues have potential to improve patient outcomes by prompting earlier and more appropriate clinical care. Future work is needed to evaluate these systems in large-scale randomized trials, to better understand the human factors driving adoption and action at the provider level, and to develop technological frameworks to facilitate broader clinical implementation. K.E.R. declares consulting fees from Janssen Pharmaceuticals .