A recent model developed at the Institute for Health Metrics and Evaluation (IHME) provides forecasts for ventilator use and hospital beds required for the care of COVID19 patients on a state-by-state basis throughout the United States over the period March 2020 through August 2020. In addition, the manuscript and associated website provide projections of deaths per day and total deaths throughout this period for the entire US, as well as for the District of Columbia. Our goal in this report is to provide a framework for evaluating the predictive validity of model forecasts for COVID19 outcomes as data become sequentially available, using the IHME prediction of daily deaths as an example. Given our goal is to provide an evaluation framework, we treat the IHME model as a black box and examine the projected numbers of deaths per day in light of the ground truth to help begin to understand the predictive accuracy of the model .
This paper provides a formal evaluation of the predictive performance of a model (and its various updates) developed by the Institute for Health Metrics and Evaluation (IHME) for predicting daily deaths attributed to COVID19 for each state in the United States. The IHME models have received extensive attention in social and mass media, and have influenced policy makers at the highest levels of the United States government. For effective policy making the accurate assessment of uncertainty, as well as accurate point predictions, are necessary because the risks inherent in a decision must be taken into account, especially in the present setting of a novel disease affecting millions of lives. To assess the accuracy of the IHME models, we examine both forecast accuracy as well as the predictive performance of the 95\% prediction intervals provided by the IHME models. We find that the initial IHME model underestimates the uncertainty surrounding the number of daily deaths substantially. Specifically, the true number of next day deaths fell outside the IHME prediction intervals as much as 70\% of the time, in comparison to the expected value of 5\%. In addition, we note that the performance of the initial model does not improve with shorter forecast horizons. Regarding the updated models, our analyses indicate that the later models do not show any improvement in the accuracy of the point estimate predictions. In fact, there is some evidence that this accuracy has actually decreased over the initial models. Moreover, when considering the updated models, while we observe a larger percentage of states having actual values lying inside the 95\% prediction intervals (PI), our analysis suggests that this observation may be attributed to the widening of the PIs. The width of these intervals calls into question the usefulness of the predictions to drive policy making and resource allocation .
R
A recent model developed at
This paper provides a formal evaluation of the predictive performance of a model (and its various updates) developed by
R
provides forecasts for ventilator use and hospital beds required for
for predicting daily deaths attributed to COVID19 for each state in the United States. The IHME models have received extensive attention in social and mass media, and have influenced policy makers at the highest levels of
D
care of COVID19 patients on a state-by-state basis throughout the
R
over the period March 2020 through August 2020. In addition, the manuscript and associated website provide projections of deaths per day and total deaths throughout this period for the entire US,
government. For effective policy making the accurate assessment of uncertainty, as well as accurate point predictions, are necessary because the risks inherent in a decision must be taken into account, especially in the present setting of a novel disease affecting millions of lives. To assess the accuracy of the IHME models, we examine both forecast accuracy
R
for the District of Columbia. Our goal in this report is to provide a framework for evaluating the predictive validity of model forecasts for COVID19 outcomes as data become sequentially available, using the IHME prediction
the predictive performance of the 95\% prediction intervals provided by the IHME models. We find that the initial IHME model underestimates the uncertainty surrounding the number
R
as an example. Given our goal is to provide an evaluation framework, we treat the IHME model as a black box and examine the projected numbers of deaths per day in light of the ground truth to help begin to understand the predictive accuracy of the model
substantially. Specifically, the true number of next day deaths fell outside the IHME prediction intervals as much as 70\% of the time, in comparison to the expected value of 5\%. In addition, we note that the performance of the initial model does not improve with shorter forecast horizons. Regarding the updated models, our analyses indicate that the later models do not show any improvement in the accuracy of the point estimate predictions. In fact, there is some evidence that this accuracy has actually decreased over the initial models. Moreover, when considering the updated models, while we observe a larger percentage of states having actual values lying inside the 95\% prediction intervals (PI), our analysis suggests that this observation may be attributed to the widening of the PIs. The width of these intervals calls into question the usefulness of the predictions to drive policy making and resource allocation