Caution Warranted: Using the Institute for Health Metrics and Evaluation Model for Predicting the Course of the COVID-19 Pandemic
Authors: Nicholas P. Jewell, Joseph A. Lewnard, Britta L. Jewell
Journal: Annals of Internal Medicine (2020)

Ideas and Opinions14 April 2020Caution Warranted: Using the Institute for Health Metrics and Evaluation Model for Predicting the Course of the COVID-19 PandemicFREENicholas P. Jewell, PhD, Joseph A. Lewnard, PhD, and Britta L. Jewell, PhDNicholas P. Jewell, PhDLondon School of Hygiene & Tropical Medicine, London, United Kingdom, and University of California, Berkeley, Berkeley, California (N.P.J.)Search for more papers by this author, Joseph A. Lewnard, PhDUniversity of California, Berkeley, Berkeley, California (J.A.L.)Search for more papers by this author, and Britta L. Jewell, PhDImperial College London, London, United Kingdom (B.L.J.)Search for more papers by this authorAuthor, Article, and Disclosure Informationhttps://doi.org/10.7326/M20-1565 SectionsAboutVisual AbstractPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail A recent modeling analysis by the Institute for Health Metrics and Evaluation (IHME) (1) projecting deaths due to coronavirus disease 2019 (COVID-19) has attracted considerable attention, including from the U.S. government (2). The model used COVID-19 mortality projections to estimate hospital bed requirements and deaths. We agree with qualitative conclusions that demand for hospital beds may exceed capacity and efforts to enhance mitigation policies and surge planning are essential. Data endorse shelter-in-place orders and suggest that these measures must remain while awaiting advances in surveillance, treatment, and vaccines.The IHME projections are based not on transmission dynamics but on a statistical model with no epidemiologic basis. Specifically, the model used reported worldwide COVID-19 deaths and extrapolated similar patterns in mortality growth curves to forecast expected deaths. The technique uses mortality data, which are generally more reliable than testing-dependent confirmed case counts. Outputs suggest precise estimates (albeit with uncertainty bounds) for all regions until the epidemic ends. This appearance of certainty is seductive when the world is desperate to know what lies ahead. However, the underlying data and statistical model must be interpreted cautiously. Here, we raise concerns about the validity and usefulness of the projections for policymakers.First, the statistical model assumes that systematic variation in mortality curves across regions is captured by timing of social distancing decisions and that other differences are explained by random effects. The model rests on the likely incorrect assumption that effects of social distancing policies are the same everywhere and that suppression policies will be implemented in all regions and will remain effective throughout.These projections may suffer from the fallacy of Farr's law, a similar nonmechanistic method in which epidemics are assumed to follow a normal distribution shifted and scaled to fit data. However, epidemic curves fit early data in multiple ways that affect expected durations or maximum numbers of deaths observed (3, 4). This is important as epidemics progress and deviations from normal distributions are expected—for example, due to “second waves” after interventions are eased.Second, the approach models mortality curves for every region with parameters for the final total, the pace of mortality growth, and timing of when the growth curve inflects. We have observed few entire curves; Hubei province's is the most complete, and curves in regions of Italy and in South Korea have subsequently passed their peaks. After age and subtle differences in policy timings are accounted for, all curves are assumed to follow these general patterns. This is optimistic: China enacted more stringent restrictions than elsewhere after observing only 17 deaths (5), and South Korea benefited from widespread testing to isolate cases early. Updating results may diminish the extent to which inference depends on a few settings, but countries that have flattened death curves earliest may not provide a basis for extrapolating trends in areas where similar control could prove elusive. Moreover, recrudescence of transmission remains possible between transient intervention periods.Third, death counts can be unreliable and reporting differences occur even within regions. Although Hubei province data represent the most complete mortality curve available, these numbers are suspect (6). Italy and other countries only report hospital deaths, thus neglecting deaths elsewhere (7). Lack of testing may impede deaths from being attributed to COVID-19, particularly initially. For example, in Bergamo, Italy, the number of anomalous deaths is several times larger than the official COVID-19 numbers (8). Reporting delays understate the growth of mortality curves initially, which is particularly concerning because the model uses early patterns for future projections. The recent addition of aggregate hospitalizations is also problematic due to inconsistent and poor reporting.Although undercounting of deaths affects the final epidemic size, it may not affect shapes of mortality curves. However, this assumes that undercounting and reporting delays are similar across time and geography, with variations captured by posited random effects. Such reporting issues resemble those arising commonly in infectious disease surveillance analyses and should be accounted for statistically.Fourth, despite not accounting for data and model structure flaws, uncertainty bands are broad. If all sources of uncertainty were accommodated, confidence intervals would necessarily be wider, making projections less proscriptive for policy decisions. Unaccounted sources of uncertainty arise from inaccurate temporal data on mortality and hospitalization counts; model misspecification, including parametrization choices; and inaccuracies in assumptions regarding the timing and effect of social distancing policies across regions. Graphical representation of uncertainty in curves is also not conducive to understanding uncertainty in peak daily death or hospital admission dates. This uncertainty would be more evident if only the “envelope” of uncertainty was shown without the central curve, which currently suggests greater precision than the model is able to offer.Fifth, updated projections already reveal substantial volatility. For New York, the model predicted 10 243 deaths (range, 5167 to 26 444) on 27 March and 15 546 (range, 8016 to 22 255) on 30 March. Given the opaqueness of the model and underlying source data, it is challenging to understand why other regions' projections also change dramatically. The alignment of past predictions with reality and current predictions should also be reported transparently.Finally, the projections are being interpreted misleadingly in formal and social media, without sufficient caveats, and outcomes differ substantially from those of other models (9, 10). Upper uncertainty bounds are being interpreted as “worst-case” when, at best, they reflect only one scenario. The model yields an attack rate less than 5%; higher rates will lead to greater mortality than upper bounds.Ultimately, IHME's model may be reliable only for short-term projections. For hospital demand projections, patient-level clinical outcome data will enable more accurate conclusions than poorly reported worldwide aggregate mortality data with point estimates of how deaths translate into hospital use. Local data are less likely to be subject to undercounting or reporting errors, helping hospitals better prepare for the immediate future. It is also unlikely that a “one-size” model will fit all regions at all times. Policymakers will be best served when they consider projections from multiple models, thus increasing the understanding of factors that influence disparate projections and enhancing comprehension of unaccounted uncertainty in any one model. Major policy decisions need model input, but models are valuable only to the extent that outputs are transparent, are valid, are based on accurate documented sources, are rigorously evaluated, and yield robust and reliable projections.References1. Institute for Health Metrics and Evaluation. New COVID-19 forecasts: US hospitals could be overwhelmed in the second week of April by demand for ICU beds, and US deaths could total 81,000 by July. Accessed at www.healthdata.org/news-release/new-covid-19-forecasts-us-hospitals-could-be-overwhelmed-second-week-april-demand-icu on 30 March 2020. Google Scholar2. Institute for Health Metrics and Evaluation (IHME). IHME | The Chris Murray Model | Dr. Deborah Birx | COVID-19 Model | “Meet The Press”. 31 March 2020. Accessed at www.youtube.com/watch?v=vnd_wlTL8_4 on 3 April 2020. Google Scholar3. Bregman DJ, Langmuir AD. Farr's law applied to AIDS projections. JAMA. 1990;263:1522-5. [PMID: 2308183] doi:10.1001/jama.1990.03440110088033 CrossrefMedlineGoogle Scholar4. @CT_Bergstrom. 1. A somewhat more technical thread than usual. 28 March 2020. Accessed at https://twitter.com/CT_Bergstrom/status/1243819232950751233 on 13 April 2020. Google Scholar5. Kuo L. Coronavirus: China bars 11m residents from leaving city at centre of outbreak. The Guardian. 22 January 2020. Accessed at www.theguardian.com/world/2020/jan/22/coronavirus-china-measures-rein-spread-mutate-disease-death-toll on 31 March 2020. Google Scholar6. Long Q, Siu-fung L, Mudie L. Estimates Show Wuhan Death Toll Far Higher Than Official Figure. Radio Free Asia. 27 March 2020. Accessed at www.rfa.org/english/news/china/wuhan-deaths-03272020182846.html on 31 March 2020. Google Scholar7. Stancati M, Sylvers E. Italy's Coronavirus Death Toll Is Far Higher Than Reported. The Wall Street Journal. 1 April 2020. Accessed at www.wsj.com/articles/italys-coronavirus-death-toll-is-far-higher-than-reported-11585767179 on 4 April 2020. Google Scholar8. Foresti CCL. The real death toll for Covid-19 is at least 4 times the official numbers. Corriere Della Serra. 26 March 2020. Accessed at www.corriere.it/politica/20_marzo_26/the-real-death-toll-for-covid-19-is-at-least-4-times-the-official-numbers-b5af0edc-6eeb-11ea-925b-a0c3cdbe1130.shtml?refresh_ce-cp on 31 March 2020. Google Scholar9. Ferguson NM, Laydon D, Nedjati-Gilani G, et al. Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand. Imperial College London; 16 March 2020. Accessed at www.imperial.ac.uk/media/imperial-college/medicine/mrc-gida/2020-03-16-COVID19-Report-9.pdf on 31 March 2020. Google Scholar10. Kerr C, Hagedorn B, Mistry D, et al. Working paper: Projected COVID-19 epidemic trends and health system needs for Oregon. Institute for Disease Modeling; 23 March 2020. Accessed at https://covid.idmod.org/data/Projected_COVID19_epidemic_trends_health_system_needs_Oregon.pdf on 31 March 2020. Google Scholar Comments 0 Comments Sign In to Submit A Comment Jerome S. Tannenbaum, M.D. Ph.D., FACPSanderling Dialysis Clinics21 April 2020 Regulatory influence may influence the predictive model The authors have raised a number of valid reasons to be careful when interpreting the output of the IHME model. In addition to the reasons cited by the authors, the output may also be skewed as a result of unintended incentives for hospitals to over-report COVID, in an effort to comply with the stipulations of the recent CARES Act Grants. The CARES Act Grants stipulate that the funds must be used for COVID-related services. Hospitals have no incentive to sort out whether a patient, who was infected with COVID-19, died of complications from an underlying condition, or whether COVID-19 induced the decompensation and complications of the patient’s underlying disease. In other words, hospitals may be including morbidity and mortality events even in situations where COVID-19 is simply an incidental finding. Author, Article, and Disclosure InformationAuthors: Nicholas P. Jewell, PhD; Joseph A. Lewnard, PhD; Britta L. Jewell, PhDAffiliations: London School of Hygiene & Tropical Medicine, London, United Kingdom, and University of California, Berkeley, Berkeley, California (N.P.J.)University of California, Berkeley, Berkeley, California (J.A.L.)Imperial College London, London, United Kingdom (B.L.J.)Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M20-1565.Corresponding Author: Nicholas P. Jewell, PhD, Department of Medical Statistics, London School of Hygiene & Tropical Medicine, Keppel Street, London WC1E 7HT, United Kingdom; e-mail, jewell@berkeley.edu.Current Author Addresses: Dr. N.P. Jewell: Department of Medical Statistics, London School of Hygiene & Tropical Medicine, Keppel Street, London WC1E 7HT, United Kingdom.Dr. Lewnard: Division of Epidemiology and Biostatistics, School of Public Health, University of California, Berkeley, Room 5410, 2121 Berkeley Way West, Berkeley, CA 94720.Dr. B.L. Jewell: Department of Infectious Disease Epidemiology, Imperial College London, Norfolk Place, London W2 1PG, United Kingdom.Author Contributions: Conception and design: N.P. Jewell, J.A. Lewnard, B.L. Jewell.Analysis and interpretation of the data: B.L. Jewell.Drafting of the article: N.P. Jewell, B.L. Jewell.Critical revision of the article for important intellectual content: N.P. Jewell, J.A. Lewnard, B.L. Jewell.Final approval of the article: N.P. Jewell, J.A. Lewnard, B.L. Jewell.Statistical expertise: N.P. Jewell.Collection and assembly of data: B.L. Jewell.This article was published at Annals.org on 14 April 2020. PreviousarticleNextarticle Advertisement FiguresReferencesRelatedDetails Metrics Cited byEstimating Covid-19 transmission time using Hawkes point processesAn EEG-based marker of functional connectivity: detection of major depressive disorderGlobal Health Interventions: The Military, the Magic Bullet, the Deterministic Model—and Intervention OtherwiseAn Automated Lightweight Deep Neural Network for Diagnosis of COVID-19 from Chest X-ray ImagesProfessional ethics for infectious disease control: moral conflict management in modern public health practiceMulti-Disease Prediction Techniques using Machine and Deep LearningHidden Costs of the COVID-19 Pandemic ResponsePassengers’ Intentions to Use Public Transport during the COVID-19 Pandemic: A Case Study of Bangkok and JakartaStudy of optimal vaccination strategies for early COVID-19 pandemic using an age-structured mathematical model: A case study of the USAMulti-model-Based Decision Support in Pandemic ManagementNumbers and MeaningA Bayesian susceptible-infectious-hospitalized-ventilated-recovered model to predict demand for COVID-19 inpatient care in a large healthcare systemEstimates of excess mortality for the five Nordic countries during the COVID-19 pandemic 2020−2021Evaluating the Comparative Accuracy of COVID-19 Mortality Forecasts: An Analysis of the First-Wave Mortality Forecasts in the United StatesA Study on Predicting the Outbreak of COVID-19 in the United Arab Emirates: A Monte Carlo Simulation ApproachAccounting for super-spreader events and algebraic decay in SIR modelsOn the use of the Reproduction Number for SARS-COV-2: Estimation, Misinterpretations and Relationships with other Ecological MeasuresOptimal lockdowns for COVID‐19 pandemics: Analyzing the efficiency of sanitary policies in EuropeCOVID-19 infection is a major cause of acute telogen effluviumCOVID-19 and the Path ForwardNonlinear Modal Regression for Dependent Data with Application for Predicting Covid-19Infectious Disease Modeling: Recommendations for Public Health Decision-MakersModeling for the Stringency of Lock-Down Policies: Effects of Macroeconomic and Healthcare Variables in Response to the COVID-19 PandemicPurposes and duties in scientific modellingMitigation Planning and Policies Informed by COVID-19 Modeling: A Framework and Case Study of the State of HawaiiEvolution of Select Epidemiological Modeling and the Rise of Population Sentiment Analysis: A Literature Review and COVID-19 Sentiment IllustrationTransmission dynamics model and the coronavirus disease 2019 epidemic: applications and challengesMulti-model-Based Decision Support in Pandemic ManagementSub-epidemic Model Forecasts During the First Wave of the COVID-19 Pandemic in the USA and European HotspotsRole of Artificial Intelligence in Diagnosis of Infectious DiseasesDesign and operation of healthcare facilities using batch-lines: the COVID-19 case in QatarStatistical Deconvolution for Inference of Infection Time SeriesDescriptive understanding and prediction in COVID-19 modellingA simple method to describe the COVID-19 trajectory and dynamics in any country based on Johnson cumulative density function fittingPredicting Hospital Resource Use During COVID-19 Surges: A Simple but Flexible Discretely Integrated Condition Event Simulation of Individual Patient-Hospital TrajectoriesIdentifiability and predictability of integer- and fractional-order epidemiological models using physics-informed neural networksEliciting Beliefs about COVID-19 Prevalence and Mortality: Epidemiological Models Compared with The StreetStatistical Models for COVID-19 Incidence, Cumulative Prevalence, and R tA model for the spread of infectious diseases compatible with case dataSetting the boundaries of COVID-19 lockdown relaxation measuresSpatial analysis of COVID-19 and traffic-related air pollution in Los AngelesAccounting for uncertainty during a pandemicModeling Epidemic Spread among a Commuting Population Using Transport SchemesQuantifying Mobility and Mixing Propensity in the Spatiotemporal Context of a Pandemic SpreadAnalysis of Key Factors of a SARS-CoV-2 Vaccination Program: A Mathematical Modeling ApproachImpact of a New SARS-CoV-2 Variant on the Population: A Mathematical Modeling ApproachOn the role of statisticians and modelers in responding to AIDS and COVID‐19Commentary on the role of statisticians in pandemicsThe pitfalls of modelling the effects of COVID-19 on gender-based violence: lessons learnt and ways forwardCoordinated Strategy for a Model-Based Decision Support Tool for Coronavirus Disease, Utah, USAForecasting COVID-19 pandemic: Unknown unknowns and predictive monitoringCOVID-19–specific skin changes related to SARS-CoV-2: Visualizing a monumental public health challengeHow Many SARS-CoV-2–Infected People Require Hospitalization? Using Random Sample Testing to Better Inform Preparedness EffortsSARS‐CoV‐2 in Germany: Association between deaths and positive rateThe Quixotic Task of Forecasting Peaks of COVID-19: Rather Focus on Forward and Backward ProjectionsEpistemic Corruption, the Pharmaceutical Industry, and the Body of Medical ScienceBreakthrough healthcare technologies in the COVID-19 era: a unique opportunity for cardiovascular practitioners and patientsMapping COVID-19 in Context: Promoting a Proportionate Perspective on the PandemicPromoting the Resilience of the Italian Population Against SARS-CoV-2Using Demographic Pattern Analysis to Predict COVID-19 Fatalities on the US County LevelEffect of Timing of and Adherence to Social Distancing Measures on COVID-19 Burden in the United States A Simulation Modeling ApproachOguzhan Alagoz, PhD, Ajay K. Sethi, PhD, MHS, Brian W. Patterson, MD, MPH, Matthew Churpek, MD, MPH, PhD, and Nasia Safdar, MD, PhDChallenges and Limitations of Geospatial Data and Analyses in the Context of COVID-19Development and dissemination of infectious disease dynamic transmission models during the COVID-19 pandemic: what can we learn from other pathogens and how can we move forward?Redefining Medicine's Relationship With the Media in the Era of COVID-19Decision support tool for hospital resource allocation during the COVID-19 pandemicPiece-wise linear regression: A new approach to predict COVID-19 spreadingRole of Machine Learning Techniques to Tackle the COVID-19 Crisis: Systematic ReviewEvaluation of a Parsimonious COVID-19 Outbreak Prediction Model: Heuristic Modeling Approach Using Publicly Available Data SetsImpact of Relaxing Covid-19 Social Distancing Measures on Rural North Wales: A Simulation AnalysisPredictive accuracy of a hierarchical logistic model of cumulative SARS-CoV-2 case growth until May 2020Managing COVID-19 within and across health systems: why we need performance intelligence to coordinate a global responseHistory of global burden of disease assessment at the World Health OrganizationSubstantial underestimation of SARS-CoV-2 infection in the United StatesEpidemiology and transmission dynamics of COVID-19 in two Indian statesEstimating Shortages in Capacity to Deliver Continuous Kidney Replacement Therapy During the COVID-19 Pandemic in the United StatesA safe level of alcohol consumption: the right answer demands the right questionRole of intelligent computing in COVID-19 prognosis: A state-of-the-art reviewCOVID‐19 with dermatologic manifestations and implications: An unfolding conundrumForecasting efforts from prior epidemics and COVID-19 predictionsResponding to the Challenges of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)A Review of Multi‐Compartment Infectious Disease ModelsWrong but Useful — What Covid-19 Epidemiologic Models Can and Cannot Tell UsEarly network properties of the COVID-19 pandemic – The Chinese scenarioThousands Of Lives Could Be Saved In The US During The COVID-19 Pandemic If States Exchanged VentilatorsIncidence, clinical outcomes, and transmission dynamics of severe coronavirus disease 2019 in California and Washington: prospective cohort studyCOVID-19: What We Know So FarThe Benefits of Coronavirus Suppression: A Cost-Benefit Analysis of the Response to the First Wave of COVID-19Dynamic Panel Estimate–Based Health Surveillance of SARS-CoV-2 Infection Rates to Inform Public Health Policy: Model Development and Validation 4 August 2020Volume 173, Issue 3 Page: 226-227 Keywords COVID-19 Forecasting Hospitalizations Hygiene Information storage and retrieval Mortality Normal distribution Public policy Social distancing Statistical models ePublished: 14 April 2020 Issue Published: 4 August 2020 Copyright & PermissionsCopyright © 2020 by American College of Physicians. All Rights Reserved.PDF downloadLoading ...