Evaluation and communication of pandemic scenarios
Authors: Philip Gerlee, Henrik Thorén, Anna Saxne Jöud, Torbjörn Lundh, Armin Spreco, Anders Nordlund, Thomas Brezicka, Tom Britton, Magnus Kjellberg, Henrik Källberg, Anders Tegnell, Lisa Brouwers, Toomas Timpka
Journal: The Lancet Digital Health (2024)

In recent years, publications in The Lancet Digital Health have presented research involving pandemic scenarios.1Nixon K Jindal S Parker F et al.Real-time COVID-19 forecasting: challenges and opportunities of model performance and translation.Lancet Digital Health. 2022; 4: e699-e701Summary Full Text Full Text PDF PubMed Scopus (6) Google Scholar However, during the early stages of the COVID-19 pandemic, the terms prediction, scenario, and forecast were often used interchangeably as discussed by Kristen Nixon and colleagues,1Nixon K Jindal S Parker F et al.Real-time COVID-19 forecasting: challenges and opportunities of model performance and translation.Lancet Digital Health. 2022; 4: e699-e701Summary Full Text Full Text PDF PubMed Scopus (6) Google Scholar leading to confusion. Although distinctions between these concepts have been refined during the pandemic,2Howerton E Contamin L Mullany LC et al.Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty.Nat Commun. 2023; 147260Crossref PubMed Scopus (16) Google Scholar we find that clarification is needed for the use of scenario projections as narrative devices. We also encourage more discussion regarding the terminology used to describe scenarios and how they are evaluated. What is the distinction between the terms and why is the distinction important? Underlying all the distinctions are mathematical models that, together with numerical values of their parameters (eg, the rate of transmission), are solved numerically to generate outputs describing the future. These descriptions of future states of a system are usually called predictions. The term forecast has been used to denote an unconditional prediction about what will happen in the future.3Schroeder SA How to interpret COVID-19 predictions: reassessing the IHME's model.Philosophy of Medicine. 2021; 2: 1-7Google Scholar By contrast, a scenario projection is a conditional prediction—ie, a prediction that is conditional on a set of assumptions about the future (ie, a scenario). Forecasts are typically short term (less than a month) because prediction uncertainty often makes them functionally useless on longer time scales, whereas scenario projections are medium to long term. Although separation of the terms has been seen as important, the difference is not clear-cut.4Winsberg E Harvard S Purposes and duties in scientific modelling.J Epidemiol Community Health. 2022; 76: 512-517Crossref Scopus (13) Google Scholar All models contain some assumptions and idealisations that need to be considered when assessing the validity of the model. One purpose of pandemic forecasts in public health is to inform about expected disease incidence and to support allocation of health-care resources. Such forecasts were deemed useful in many countries during the COVID-19 pandemic, especially for decision support at local or regional levels.5Fox SJ Lachmann M Tec M et al.Real-time pandemic surveillance using hospital admissions and mobility data.Proc Natl Acad Sci USA. 2022; 119e2111870119Crossref Scopus (29) Google Scholar Scenario projections, on the other hand, were used for a multitude of purposes during the pandemic—eg, assessing the severity of outbreaks, estimating the effects of different vaccination strategies and non-pharmaceutical interventions, and for outlining worst-case scenarios for hospital bed demand.6Borchering RK Viboud C Howerton E et al.Modeling of future COVID-19 cases, hospitalizations, and deaths, by vaccination rates and nonpharmaceutical intervention scenarios—United States, April–September 2021.MMWR Morb Mortal Wkly Rep. 2021; 70: 719Crossref PubMed Scopus (106) Google Scholar From a public health-policy perspective, scenario projections generated from models serve predominately as virtual testbeds for exploring chains of events probable to occur given a set of assumptions such as future rates of disease transmission given an assumed rate of vaccine roll-out. A taxonomy of scenario design was recently put forward based on an analogy with experimental design.7Runge MC Shea K Howerton E et al.Scenario design for infectious disease projections: integrating concepts from decision analysis and experimental design.Epidemics. 2024; 47100775Crossref Scopus (1) Google Scholar By considering scenario design along two independent axes—intervention and uncertainty—they identify six independent classes of design, such as sensitivity analysis, situational awareness, and horizon scanning. However, during the COVID-19 pandemic, scenario projections also served another important purpose, namely as narrative devices. For example, during spring 2020, many governments appealed to the public to impose social distancing for flattening the curve using conditional predictions (ie, scenario projections) as backdrops. These projections could, for instance, contrast hospital admissions in the absence of social distancing with admission rates conditional to social distancing, observing that hospital admissions conditional to social distancing fell below the critical capacity threshold. The purpose of such scenario projections goes beyond illustrating possible future states of the world. Rather, the purpose is to implore presumptive audiences to act in a specific way. Although this purpose is related to Runge and colleagues' concept of 'decision making' design, their taxonomy does not cover this persuasive aspect.7Runge MC Shea K Howerton E et al.Scenario design for infectious disease projections: integrating concepts from decision analysis and experimental design.Epidemics. 2024; 47100775Crossref Scopus (1) Google Scholar Of note, the use of a projection as narrative device is not always clear and some projections reported in preprints during the early phases of the pandemic could be interpreted as calls for heavier restrictions (eg, Gardner and colleagues),8Gardner JM Willem L Van Der Wijngaart W et al.Intervention strategies against COVID-19 and their estimated impact on Swedish healthcare capacity.Int J Epidemiol. 2020; 49: 1443-1453PubMed Google Scholar thus had a transactional aim without clearly expressing this. The persuasive aspects of scenario projections have been described using terms such as performativity and interactive effects, which refer to the ability of projections to have an effect on future states of the world.9Oomen J Hoffman J Hajer MA Techniques of futuring: on how imagined futures become socially performative.Eur J Soc Theory. 2022; 25: 252-270Crossref Scopus (114) Google Scholar Here, modelers (and decision makers) have an important responsibility when from an infinite set of possible scenario projections, they pick a handful that are simulated and communicated. These choices can be used to control the narrative that is communicated and have a profound effect on how a contingency unfolds. Given the wide range of applications of scenario projections in pandemic modelling, the presentation of the model output should align with its purpose. Modelers and decision makers should be explicit about the underlying model assumptions and, additionally for scenario projections, it should be clear what future events the results are conditioned on and the intended purpose of the scenario design. Evaluating the usefulness or performance of scenario projections is more complex compared with forecasts. Reporting of forecasting model performance can be evaluated by comparing the prediction with the actual outcome using formal metrics (eg, the mean absolute percentage error or weighted interval score for probabilistic forecasts).10Cramer EY Ray EL Lopez VK et al.Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.Proc Natl Acad Sci USA. 2022; 119e2113561119Crossref Scopus (126) Google Scholar Projections, on the other hand, cannot straight-forwardly be compared with actual outcomes. Formal evaluations of conditional predictions can be performed using post-hoc information on the assumptions made at the time of the prediction and the real-world outcomes. A framework for such evaluations was recently presented and applied in the context of the US COVID-19 Scenario Modeling Hub.2Howerton E Contamin L Mullany LC et al.Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty.Nat Commun. 2023; 147260Crossref PubMed Scopus (16) Google Scholar However, such evaluations might be difficult to implement because there are no guarantees that this information can be gathered and that the components used to build the scenario projections are sufficient to represent what later occurred in the real world. For example, if a prediction was conditioned on an increase in social distancing, it might not be possible post hoc to obtain accurate values of the actual social distancing during the considered timeframe. Nonetheless, conditional predictions that at post-hoc analysis are found to have failed to match events in the real world could still be useful when deployed as narrative devices. If adherence with recommendations of social distancing is improved using a scenario model (ie, disease transmission is reduced), then the scenario projection has fulfilled its purpose although the factual outcome is far from the projections made. The same is true for analyses of worst-case scenarios when considering the application of precautionary principles in policy making. For rational use of narrative scenarios, they need to be evaluated with regards to their intended purpose using measures and endpoints adapted to this purpose. When presenting a pandemic scenario projection, it is therefore of utmost importance to describe its purpose and the assumptions made using clearly defined and unambiguous terminology. A narrative scenario projection that communicates a worst-case estimate of the transmission and virulence of an infectious agent might in the short-term influence population behaviour and serve the purpose to reduce the spread of disease. However, if the foundations are not communicated in full, such narrative scenarios might in the long run undermine the general trust in science and public health institutions. We therefore argue that using a strict terminology when reporting and evaluating pandemic scenario projections will help prevent public distrust and facilitate scientific communication. We declare no competing interests.