Early human judgment forecasts of human monkeypox, May 2022
Authors: Thomas McAndrew, Maimuna S Majumder, Andrew A Lover, Srini Venkatramanan, Paolo Bocchini, Tamay Besiroglu, Allison Codi, David Braun, Gaia Dempsey, Sam Abbott, Sylvain Chevalier, Nikos I Bosse, Juan Cambeiro
Journal: The Lancet Digital Health (2022)

Beginning in May, 2022, clusters of monkeypox virus infection have been observed in multiple countries, including the UK, Spain, Portugal, and the USA.1CDCCDC and Health Partners Responding to Monkeypox Case in the U.S.https://www.cdc.gov/media/releases/2022/s0518-monkeypox-case.htmlDate: 2022Date accessed: May 19, 2022Google Scholar In response, the US Centers for Disease Control and Prevention and WHO have increased surveillance for orthopoxvirus infections and have asked health-care providers to be alert for patients who present with a rash-like illness that is consistent with human monkeypox.1CDCCDC and Health Partners Responding to Monkeypox Case in the U.S.https://www.cdc.gov/media/releases/2022/s0518-monkeypox-case.htmlDate: 2022Date accessed: May 19, 2022Google Scholar, 2Isidro J Borges V Pinto M et al.Multi-country outbreak of monkeypox virus: genetic divergence and first signs of microevolution.https://virological.org/t/multi-country-outbreak-of-monkeypox-virus-genetic-divergence-and-first-signs-of-microevolution/806Date: 2022Date accessed: May 23, 2022Google ScholarWe present an ongoing human judgment effort in collaboration with Metaculus—a forecasting technology platform—to make crowdsourced ensemble predictions of (1) the estimated number of reported human monkeypox infections globally as of Dec 31, 2022, (2) the estimated number of deaths due to human monkeypox globally as of Dec 31, 2022, (3) the number of monkeypox cases reported across Europe as of July 1, 2022, (4) the number of monkeypox cases reported in the USA as of July 1, 2022, (5) the number of monkeypox cases reported in Canada as of July 1, 2022, (6) the number of countries that will report at least one case of human monkeypox by July 31, 2022, (7) the number of US states that will report at least one case by July 1, 2022, and (8) whether WHO will declare the monkeypox outbreak a Public Health Emergency of International Concern (PHEIC) before Dec 31, 2022.As demonstrated during previous infectious disease outbreaks, crowdsourced human judgment can successfully estimate many different quantities that have epidemiological meaning or are of importance to public health decision makers.3Farrow DC Brooks LC Hyun S Tibshirani RJ Burke DS Rosenfeld R A human judgment approach to epidemiological forecasting.PLoS Comput Biology. 2017; 13e1005248Crossref PubMed Scopus (33) Google Scholar, 4McAndrew T Reich N G An expert judgment model to predict early stages of the COVID-19 outbreak in the United States.medRxiv. 2020; (published online Sept 23.) (preprint).https://doi.org/10.1101/2020.09.21.20196725PubMed Google Scholar, 5McAndrew T Cambeiro J Besiroglu T Aggregating human judgment probabilistic predictions of the safety, efficacy, and timing of a COVID-19 vaccine.Vaccine. 2022; 40: 2331-2341Crossref PubMed Scopus (2) Google Scholar, 6Bosse NI Abbot S Bracher J et al.Comparing human and model-based forecasts of COVID-19 in Germany and Poland.medRxiv. 2021; (published online Dec 5.) (preprint).https://doi.org/10.1101/2021.12.01.21266598Google Scholar, 7Braun D Ingram D Ingram D Khan B Marsh J McAndrew T Incorporating crowdsourced perceptions of human behavior into computational forecasts of US national incident cases of COVID-19.OSF Preprints. 2022; (published online May 5.) (preprint).https://doi.org/10.31219/osf.io/7vrmyGoogle Scholar, 8McAndrew T Codi A Cambeiro et al."Chimeric forecasting: combining probabilistic predictions from computational models and human judgment.arXiv. 2022; (published online Feb 22.) (preprint).https://doi.org/10.48550/arXiv.2202.09820PubMed Google Scholar During the 2014–15 and 2015–16 influenza seasons, crowdsourced forecasts of influenza-like illness were collected and found to be among the most accurate when compared to computational forecasts.3Farrow DC Brooks LC Hyun S Tibshirani RJ Burke DS Rosenfeld R A human judgment approach to epidemiological forecasting.PLoS Comput Biology. 2017; 13e1005248Crossref PubMed Scopus (33) Google Scholar Beginning on Feb 18, 2020, just before WHO declared COVID-19 a pandemic, experts in infectious disease modelling were called upon to generate predictions of a diverse set of quantities related to COVID-19, including the number of weekly incident cases at the US national level, the cumulative number of deaths by the end of 2020, and counterfactual values (eg, the reported number of cases at the state level under two different potential policy changes).4McAndrew T Reich N G An expert judgment model to predict early stages of the COVID-19 outbreak in the United States.medRxiv. 2020; (published online Sept 23.) (preprint).https://doi.org/10.1101/2020.09.21.20196725PubMed Google Scholar Experts also sensibly predicted the safety, timing, and efficacy of a SARS-CoV-2 vaccine 6 months before the first vaccine approval.5McAndrew T Cambeiro J Besiroglu T Aggregating human judgment probabilistic predictions of the safety, efficacy, and timing of a COVID-19 vaccine.Vaccine. 2022; 40: 2331-2341Crossref PubMed Scopus (2) Google Scholar Additional studies have found that human judgment and computational modelling might complement one another.6Bosse NI Abbot S Bracher J et al.Comparing human and model-based forecasts of COVID-19 in Germany and Poland.medRxiv. 2021; (published online Dec 5.) (preprint).https://doi.org/10.1101/2021.12.01.21266598Google Scholar, 7Braun D Ingram D Ingram D Khan B Marsh J McAndrew T Incorporating crowdsourced perceptions of human behavior into computational forecasts of US national incident cases of COVID-19.OSF Preprints. 2022; (published online May 5.) (preprint).https://doi.org/10.31219/osf.io/7vrmyGoogle Scholar, 8McAndrew T Codi A Cambeiro et al."Chimeric forecasting: combining probabilistic predictions from computational models and human judgment.arXiv. 2022; (published online Feb 22.) (preprint).https://doi.org/10.48550/arXiv.2202.09820PubMed Google ScholarWe collected 686 unique and revised predictions (358 unique and 328 revised) from May 19 to May 24, 2022, on the Metaculus human judgment crowdsourcing platform (methods, limitations, and TRIPOD reporting guidelines are in the appendix pp 15–19).The probability that the crowdsourced ensemble placed on 1000 to 10 000 estimated global infections was 0·51, with a 0·46 probability on 10 000 or more infections (figure A). The probability that the crowd placed on 100 to 1000 deaths was 0·59, with a 0·23 probability on 1000 or more deaths (figure B). Predictive quantiles for the estimated number of human monkeypox infections and deaths stabilised after the first few days of forecasting (figure C).The probability assigned that 30 to 100 countries would report one or more infections by July 31, 2022, was 0·75, with a probability of 0·09 for 100 or more countries (figure D). The probability that the crowd assigned to more than one reported infection in less than ten US states by July 1, 2022, was 0·34, in 10–20 states was 0·33, in 20–30 states was 0·20, and in 30 or more states was 0·13 (figure E). Quantiles for the number of countries predicted to report an infection increased during the forecasting period, and the median prediction of the number of US states to report an infection decreased from 35 to 14 (figure F).The crowdsourced probabilities were 0·56 for 10–100 cases, 0·37 for 100–1000 cases, and 0·03 for 1000 or more cases reported in the USA by July 1, 2022 (figure G). By the same date, the probability that the crowd assigned to less than 100 reported cases in Canada was 0·51 and to between 100 and 1000 was 0·48. However, the probability was less than 0·01 for 1000 or more cases (figure H). In Europe, the probability that the crowd placed on 500 to 1000 cases by July 1, 2022, was 0·32, and the probability placed on 2000 or more cases was 0·23 (figure I). After the first day of forecasting, the 25th and 75th percentiles for the cumulative reported cases for all three locations included a large range of values, which might represent forecasters' uncertainty in how the present outbreak will evolve (figure J).The ensemble median prediction of the probability that WHO will declare the monkeypox virus a PHEIC before Dec 31, 2022, was approximately 25%. When this question was first posted, the median prediction was above 80% and decreased within 4 days (figure K).Several limitations should be considered when aggregating human judgment predictions. Forecasts might be biased by (1) those in the crowd who might submit forecasts that are poorly calibrated; (2) anchoring effects that might act on forecasters when they view current aggregate forecasts; (3) correlations—if present—between those experts who are more likely to respond to a survey and their perceptions of the intensity of an outbreak; and (4) potential correlations between the probability that a forecaster submits a forecast and their confidence in their ability to generate an accurate prediction. Humans, like computational models, might also be biased by current surveillance data that is available during an outbreak. Additional work might focus on measuring the influence of an observable aggregate forecast on a forecast submitted by an individual and stratifying by subject matter experts and non-experts.These early results suggest that a human judgment forecasting platform can quickly generate probabilistic predictions for targets of public health importance. Predictions from a human judgment forecasting platform might be especially important when data are sparse, limiting the accuracy of statistical models, or when historical data has been collected in different locations than the present outbreak, limiting the accuracy of mechanistic models. Further, human judgment forecasts can augment epidemiological models with forecasts of policy decisions (eg, WHO declaration), which might influence epidemic trajectories. Even in dynamic situations, human judgment-based systems might aggregate information across diverse areas of expertise, disparate scientific documents (such as outbreak reports and preprints), and word-of-mouth.9Salehinejad S Jangipour Afshar P Borhaninejad V Rumor surveillance methods in outbreaks: systematic literature review.Health Promot Perspect. 2021; 11: 12-19Crossref PubMed Scopus (3) Google Scholar Quantiles of aggregated forecasts stabilised in a short period of time, suggesting a consensus which might be driven by a combination of observed data and human intuition. Human judgment forecasts therefore provide a crucial health tool to fill this gap in assessing the potential for widespread community transmission of the monkeypox virus. Beginning in May, 2022, clusters of monkeypox virus infection have been observed in multiple countries, including the UK, Spain, Portugal, and the USA.1CDCCDC and Health Partners Responding to Monkeypox Case in the U.S.https://www.cdc.gov/media/releases/2022/s0518-monkeypox-case.htmlDate: 2022Date accessed: May 19, 2022Google Scholar In response, the US Centers for Disease Control and Prevention and WHO have increased surveillance for orthopoxvirus infections and have asked health-care providers to be alert for patients who present with a rash-like illness that is consistent with human monkeypox.1CDCCDC and Health Partners Responding to Monkeypox Case in the U.S.https://www.cdc.gov/media/releases/2022/s0518-monkeypox-case.htmlDate: 2022Date accessed: May 19, 2022Google Scholar, 2Isidro J Borges V Pinto M et al.Multi-country outbreak of monkeypox virus: genetic divergence and first signs of microevolution.https://virological.org/t/multi-country-outbreak-of-monkeypox-virus-genetic-divergence-and-first-signs-of-microevolution/806Date: 2022Date accessed: May 23, 2022Google Scholar We present an ongoing human judgment effort in collaboration with Metaculus—a forecasting technology platform—to make crowdsourced ensemble predictions of (1) the estimated number of reported human monkeypox infections globally as of Dec 31, 2022, (2) the estimated number of deaths due to human monkeypox globally as of Dec 31, 2022, (3) the number of monkeypox cases reported across Europe as of July 1, 2022, (4) the number of monkeypox cases reported in the USA as of July 1, 2022, (5) the number of monkeypox cases reported in Canada as of July 1, 2022, (6) the number of countries that will report at least one case of human monkeypox by July 31, 2022, (7) the number of US states that will report at least one case by July 1, 2022, and (8) whether WHO will declare the monkeypox outbreak a Public Health Emergency of International Concern (PHEIC) before Dec 31, 2022. As demonstrated during previous infectious disease outbreaks, crowdsourced human judgment can successfully estimate many different quantities that have epidemiological meaning or are of importance to public health decision makers.3Farrow DC Brooks LC Hyun S Tibshirani RJ Burke DS Rosenfeld R A human judgment approach to epidemiological forecasting.PLoS Comput Biology. 2017; 13e1005248Crossref PubMed Scopus (33) Google Scholar, 4McAndrew T Reich N G An expert judgment model to predict early stages of the COVID-19 outbreak in the United States.medRxiv. 2020; (published online Sept 23.) (preprint).https://doi.org/10.1101/2020.09.21.20196725PubMed Google Scholar, 5McAndrew T Cambeiro J Besiroglu T Aggregating human judgment probabilistic predictions of the safety, efficacy, and timing of a COVID-19 vaccine.Vaccine. 2022; 40: 2331-2341Crossref PubMed Scopus (2) Google Scholar, 6Bosse NI Abbot S Bracher J et al.Comparing human and model-based forecasts of COVID-19 in Germany and Poland.medRxiv. 2021; (published online Dec 5.) (preprint).https://doi.org/10.1101/2021.12.01.21266598Google Scholar, 7Braun D Ingram D Ingram D Khan B Marsh J McAndrew T Incorporating crowdsourced perceptions of human behavior into computational forecasts of US national incident cases of COVID-19.OSF Preprints. 2022; (published online May 5.) (preprint).https://doi.org/10.31219/osf.io/7vrmyGoogle Scholar, 8McAndrew T Codi A Cambeiro et al."Chimeric forecasting: combining probabilistic predictions from computational models and human judgment.arXiv. 2022; (published online Feb 22.) (preprint).https://doi.org/10.48550/arXiv.2202.09820PubMed Google Scholar During the 2014–15 and 2015–16 influenza seasons, crowdsourced forecasts of influenza-like illness were collected and found to be among the most accurate when compared to computational forecasts.3Farrow DC Brooks LC Hyun S Tibshirani RJ Burke DS Rosenfeld R A human judgment approach to epidemiological forecasting.PLoS Comput Biology. 2017; 13e1005248Crossref PubMed Scopus (33) Google Scholar Beginning on Feb 18, 2020, just before WHO declared COVID-19 a pandemic, experts in infectious disease modelling were called upon to generate predictions of a diverse set of quantities related to COVID-19, including the number of weekly incident cases at the US national level, the cumulative number of deaths by the end of 2020, and counterfactual values (eg, the reported number of cases at the state level under two different potential policy changes).4McAndrew T Reich N G An expert judgment model to predict early stages of the COVID-19 outbreak in the United States.medRxiv. 2020; (published online Sept 23.) (preprint).https://doi.org/10.1101/2020.09.21.20196725PubMed Google Scholar Experts also sensibly predicted the safety, timing, and efficacy of a SARS-CoV-2 vaccine 6 months before the first vaccine approval.5McAndrew T Cambeiro J Besiroglu T Aggregating human judgment probabilistic predictions of the safety, efficacy, and timing of a COVID-19 vaccine.Vaccine. 2022; 40: 2331-2341Crossref PubMed Scopus (2) Google Scholar Additional studies have found that human judgment and computational modelling might complement one another.6Bosse NI Abbot S Bracher J et al.Comparing human and model-based forecasts of COVID-19 in Germany and Poland.medRxiv. 2021; (published online Dec 5.) (preprint).https://doi.org/10.1101/2021.12.01.21266598Google Scholar, 7Braun D Ingram D Ingram D Khan B Marsh J McAndrew T Incorporating crowdsourced perceptions of human behavior into computational forecasts of US national incident cases of COVID-19.OSF Preprints. 2022; (published online May 5.) (preprint).https://doi.org/10.31219/osf.io/7vrmyGoogle Scholar, 8McAndrew T Codi A Cambeiro et al."Chimeric forecasting: combining probabilistic predictions from computational models and human judgment.arXiv. 2022; (published online Feb 22.) (preprint).https://doi.org/10.48550/arXiv.2202.09820PubMed Google Scholar We collected 686 unique and revised predictions (358 unique and 328 revised) from May 19 to May 24, 2022, on the Metaculus human judgment crowdsourcing platform (methods, limitations, and TRIPOD reporting guidelines are in the appendix pp 15–19). The probability that the crowdsourced ensemble placed on 1000 to 10 000 estimated global infections was 0·51, with a 0·46 probability on 10 000 or more infections (figure A). The probability that the crowd placed on 100 to 1000 deaths was 0·59, with a 0·23 probability on 1000 or more deaths (figure B). Predictive quantiles for the estimated number of human monkeypox infections and deaths stabilised after the first few days of forecasting (figure C). The probability assigned that 30 to 100 countries would report one or more infections by July 31, 2022, was 0·75, with a probability of 0·09 for 100 or more countries (figure D). The probability that the crowd assigned to more than one reported infection in less than ten US states by July 1, 2022, was 0·34, in 10–20 states was 0·33, in 20–30 states was 0·20, and in 30 or more states was 0·13 (figure E). Quantiles for the number of countries predicted to report an infection increased during the forecasting period, and the median prediction of the number of US states to report an infection decreased from 35 to 14 (figure F). The crowdsourced probabilities were 0·56 for 10–100 cases, 0·37 for 100–1000 cases, and 0·03 for 1000 or more cases reported in the USA by July 1, 2022 (figure G). By the same date, the probability that the crowd assigned to less than 100 reported cases in Canada was 0·51 and to between 100 and 1000 was 0·48. However, the probability was less than 0·01 for 1000 or more cases (figure H). In Europe, the probability that the crowd placed on 500 to 1000 cases by July 1, 2022, was 0·32, and the probability placed on 2000 or more cases was 0·23 (figure I). After the first day of forecasting, the 25th and 75th percentiles for the cumulative reported cases for all three locations included a large range of values, which might represent forecasters' uncertainty in how the present outbreak will evolve (figure J). The ensemble median prediction of the probability that WHO will declare the monkeypox virus a PHEIC before Dec 31, 2022, was approximately 25%. When this question was first posted, the median prediction was above 80% and decreased within 4 days (figure K). Several limitations should be considered when aggregating human judgment predictions. Forecasts might be biased by (1) those in the crowd who might submit forecasts that are poorly calibrated; (2) anchoring effects that might act on forecasters when they view current aggregate forecasts; (3) correlations—if present—between those experts who are more likely to respond to a survey and their perceptions of the intensity of an outbreak; and (4) potential correlations between the probability that a forecaster submits a forecast and their confidence in their ability to generate an accurate prediction. Humans, like computational models, might also be biased by current surveillance data that is available during an outbreak. Additional work might focus on measuring the influence of an observable aggregate forecast on a forecast submitted by an individual and stratifying by subject matter experts and non-experts. These early results suggest that a human judgment forecasting platform can quickly generate probabilistic predictions for targets of public health importance. Predictions from a human judgment forecasting platform might be especially important when data are sparse, limiting the accuracy of statistical models, or when historical data has been collected in different locations than the present outbreak, limiting the accuracy of mechanistic models. Further, human judgment forecasts can augment epidemiological models with forecasts of policy decisions (eg, WHO declaration), which might influence epidemic trajectories. Even in dynamic situations, human judgment-based systems might aggregate information across diverse areas of expertise, disparate scientific documents (such as outbreak reports and preprints), and word-of-mouth.9Salehinejad S Jangipour Afshar P Borhaninejad V Rumor surveillance methods in outbreaks: systematic literature review.Health Promot Perspect. 2021; 11: 12-19Crossref PubMed Scopus (3) Google Scholar Quantiles of aggregated forecasts stabilised in a short period of time, suggesting a consensus which might be driven by a combination of observed data and human intuition. Human judgment forecasts therefore provide a crucial health tool to fill this gap in assessing the potential for widespread community transmission of the monkeypox virus. MSM has been awarded funds from the National Science Foundation and Boston Children's Hospital. TB is an advisor to Metaculus and holds stock options with Metaculus. SV has been awarded funds from Centers for Disease Control and Prevention, Council for State and Territorial Epidemiologists, Virginia Department of Health, National Science Foundation, Defense Threat Reduction Agency, and the University of Pittsburgh. GD holds leadership and board roles at Metaculus, which is the public benefit corporation that develops the forecasting platform used to elicit forecasts in this Comment. All other authors declare no competing interests. We would like to thank all individuals who dedicated their time and energy to submitting probabilistic forecasts of the transmission and burden of the monkeypox virus. The forecasting data used to generate figures in this Comment are available on GitHub. Real time forecasts are available on the Metaculus website. Because no personal information about forecasters was collected for this work, we determined, in consultation with an international review board analyst, that this study should not be considered Human Subjects Research. Supplementary Material Download .pdf (1.0 MB) Help with pdf files Supplementary appendix Download .pdf (1.0 MB) Help with pdf files Supplementary appendix