metadata: authors: 0: author: Andree,Bo Pieter Johannes
1: author: Chamorro Elizondo,Andres Fernando
2: author: Kraay,Aart C.
3: author: Spencer,Phoebe Girouard
4: author: Wang,Dieter
count: World
docna: 0: docna: Predicting Food Crises
docty: Policy Research Working Paper
subtopic: Public Health Promotion,Development Economics & Aid Effectiveness,Foreign Aid,Inequality,Food Security,Nutrition
teratopic: Macroeconomics and Economic Growth,Health, Nutrition and Population,Poverty Reduction,Agriculture
abstracts: cdata!: Globally, more than 130 million people
            are estimated to be in food crisis. These humanitarian
            disasters are associated with severe impacts on livelihoods
            that can reverse years of development gains. The existing
            outlooks of crisis-affected populations rely on expert
            assessment of evidence and are limited in their temporal
            frequency and ability to look beyond several months. This
            paper presents a statistical forecasting approach to predict
            the outbreak of food crises with sufficient lead time for
            preventive action. Different use cases are explored related
            to possible alternative targeting policies and the levels at
            which finance is typically unlocked. The results indicate
            that, particularly at longer forecasting horizons, the
            statistical predictions compare favorably to expert-based
            outlooks. The paper concludes that statistical models
            demonstrate good ability to detect future outbreaks of food
            crises and that using statistical forecasting approaches may
            help increase lead time for action.
repnme: repnme: Predicting Food Crises
display_title: Predicting Food Crises
listing_relative_url: /research/2020/09/32425033/predicting-food-crises
topicv3: Food Security m1326439 199,Poverty m1326687 695,Pandemics m1327520 2167,Humanitarian Aid m1503719 3039,Malnutrition m1627088 4737
publishtoextweb_dt: 2020-09-22T00:00:00Z
historic_topic: Macroeconomics and Economic Growth,Health, Nutrition and Population,Poverty Reduction,Agriculture
no_of_pages: 35
author_keys: 0: author_key: 1539948
1: author_key: 1538661
2: author_key: 123715
3: author_key: 1539158
4: author_key: 1624437
model_extractions: Policy Research Working Paper 9412

# Predicting Food Crises

_Bo Pieter Johannes Andrée Andres Chamorro Aart Kraay Phoebe Spencer Dieter Wang_ Fragility, Conflict and Violence Global Theme & Development Economics Vice Presidency September 2020
Policy Research Working Paper 9412

# **Abstract**

Globally, more than 130 million people are estimated to be in food crisis. These humanitarian disasters are associated with severe impacts on livelihoods that can reverse years of development gains. The existing outlooks of crisis-affected populations rely on expert assessment of evidence and are limited in their temporal frequency and ability to look beyond several months. This paper presents a statistical forecasting approach to predict the outbreak of food crises with sufficient lead time for preventive action. Different use cases are explored related to possible alternative targeting policies and the levels at which finance is typically unlocked. The results indicate that, particularly at longer forecasting horizons, the statistical predictions compare favorably to expert-based outlooks. The paper concludes that statistical models demonstrate good ability to detect future outbreaks of food crises and that using statistical forecasting approaches may help increase lead time for action.

This paper is a product of the Fragility, Conflict and Violence Global Theme and the Development Economics Vice Presidency. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http:// www.worldbank.org/prwp. The authors may be contacted at bandree@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Predicting Food Crises

Bo Pieter Johannes Andr ́ee<sup>a,*</sup> , Andres Chamorro<sup>a</sup> , Aart Kraay<sup>a</sup> , Phoebe Spencer<sup>a</sup> , and Dieter Wang<sup>a</sup>

**Keywords:** Famine, Food Insecurity, Extreme Events, Unbalanced Data, Costsensitive learning.

**JEL:** C01, C14, C25, C53, O10

_a ∗ World Bank, email: bandree@worldbank.org_ . This work was prepared as background for the Famine Action Mechanism (FAM). Support from the State and Peace-Building (SPF) Trust Fund (Grants N. TF0A7049 and TF0A5070 ) is gratefully acknowledged. The authors would like to thank Nadia Piffaretti, Zacharey Carmichael, Harun Dogo, Arif Hussain, Luca Russo, Jos ́e Lopez, Colin Bruce, Nick Haan, Frank Davenport, Dan Maxwell, Joanna Macrae, Soomin Park, Marco Zambotti, Sardar Azari, Therese Norman-Monroe, Jacob LaRiviere, and the IPC, WFP mVAM, and FAO teams for invaluable contributions in the initial phase of this work. In particular, we’d like to thank the participants of the FAM Workshop in Geneva on February 2018 hosted by ICRC, Artemis Working Days in Rome on April 2018 hosted by WFP, the FAM Data and Analytics meetings with global tech partners in Rome and New York on September 2018, and the participants to the Predictive Analytics workshop hosted by UN OCHA, at the Center for Humanitarian data in the Hague in April 2019. This paper reflects the views of the authors, and does not reflect the official views of the World Bank, its Executive Directors, or the countries they represent.
# **1 Introduction**

Despite progress in reducing poverty in recent decades (World Bank Group, 2018), one in nine people in the world faces hunger (FAO et al., 2019). More than 130 million people are currently estimated to be in food crisis (Food Security Information Network, 2020), meaning that they are able to meet minimum dietary needs only through irreversible coping strategies such as liquidating livelihood assets.

Food crises reflect the complex interactions of conflict, poverty, extreme weather, climate, and food price shocks (Misselhorn, 2005; Headey, 2011; Singh, 2012; D’Souza and Jolliffe, 2013), that compound in the presence of long-standing structural factors (Maxwell and Fitzpatrick, 2012). These humanitarian disasters are characterized by high levels of acute malnutrition that lead to mortality in vulnerable populations.<sup>1</sup> Beyond immediate loss of life, food crises have long-lasting consequences for survivors as well, including inter-generational health and education effects (Galler and Barrett, 2001; Veenendaal et al., 2013; Galler and Rabinowitz, 2014; Asfaw, 2016; Li et al., 2017). Recognizing these costs, the international community has long responded to food crises with humanitarian aid. In recent years, this has been complemented with a growing emphasis on investment in prevention, since in many cases it is more cost effective to prevent crises than to respond to them (Meerkatt et al., 2015; Mechler, 2016).

Targeted prevention policies require a capacity to predict when, where and how food crises will emerge, ideally well in advance (Kleinberg et al., 2015). To contribute to this capacity, this paper explores statistical approaches to predicting food crisis. Statistical models can offer systematized insight into the timing and location of possible future events, using readily-observable and verifiable data. Moreover, model-based forecasts can assign meaningful probabilities to future events, allowing expected costs, expected benefits and uncertainty to be quantified.

In the context of food crisis prediction, balancing false positives (predicting food crises when they do not occur) and false negatives (failing to predict food crises that do occur) is particularly important. Outbreaks of food crises are relatively rare events, but when they occur they are persistent. Failing to act early and prevent these protracted humanitarian disasters comes at a high human cost. On the other hand, scarce humanitarian resources limit the scope for responding to false positives, and hard choices must be made between investing in competing options for prevention, including saving funds to address future crises. These difficulties are compounded by the fact that in order to be useful, preventive measures must be taken well before all relevant information is available.

In this paper, we explore how formal statistical forecasting models can help to address these challenges. First, we generate monthly predictions at the sub-national level, optimizing for alternative targeting policies that assume different costs of false negatives relative to false positives. Second, because finance is often unlocked at a country level, predictions are developed for the total number of people in crisis-affected districts in a country. The two predictions can be derived from the same model and work in tandem: first highlighting the

> 1Black et al. (2013) conclude that almost half of child deaths globally are associated with undernutrition and Devereux (2000) estimates that deaths due to famines in the 20th century equal those of World Wars I and II combined.

2
countries that are in need; then helping to allocate finance toward the districts that face the highest risks. The empirical application uses 10 years of sub-annual assessments on district-level food insecurity outcomes in 21 developing countries together with monthly covariates that capture known drivers of food insecurity. The predictions are generated using a Random Forest and a novel approach is deployed to adjust probabilistic predictions to optimize the forecasting objectives of the paper. The forecasts are validated, up to a full year ahead, against holdout data and the accuracy is compared to historical non-model-based outlooks. The results indicate that, particularly at longer forecasting horizons, model-based predictions compare favorably. The paper concludes that statistical models demonstrate good ability to detect future crisis outbreaks for a set level of tolerance to false alarms and that using them may help increase lead time for action.

This research contributes to understanding the ability of statistical models to anticipate future food crisis in the particular context supporting early interventions. The paper builds on several earlier efforts.<sup>2</sup> Mellor (1986) discussed prevention strategies with an emphasis on economic weakness, crop failure, and subsequent price signals as leading indicators of famine. Price signals have been discussed further in particular by Seaman and Holt (1980), and their statistical significance has been investigated empirically in the context of the Ethiopian famine of 1972-1974 (Cutler, 1984) and the famine of 1984-1985 in Niger (Khan, 1994). Seaman (2000) explored a dynamic modeling approach to assess risks based on rapid household survey data and simulations of coping strategies under income and food supply shocks. More recently, researchers have begun to use machine learning techniques for prediction. For example, Mwebaze et al. (2010); Okori and Obua (2011) presented attempts at predicting household famine in Uganda between 2004 and 2005 using household data.

The importance of using systematic approaches to predicting food crises and allocating humanitarian resources effectively is likely to increase in coming years as key drivers of food insecurity are expected to worsen into the 21st century. Historical achievements in eradicating poverty have largely coincided with substantial degradation of environments (Stern et al., 1996; Andr ́ee et al., 2019). Degrading environments, climatic extremes, and growing populations will continue to put pressure on future agricultural systems (Grainger, 1990; Ingram et al., 2010; Myers et al., 2017; Diogo et al., 2017). These developments place obstacles in the path to zero hunger, specifically for the rural poor that rely on local natural assets for income and food consumption (Duraiappah, 1998; Barrett and Bevis, 2015; Barbier and Hochard, 2018). A further benefit of model-based predictions of food insecurity is that they offer an alternative to the allocating of scarce resources based on qualitative assessments, instead relying on readily-verifiable and openly available data in a systematic way to predict the deterioration in food security when action is delayed.

The paper is structured as follows. Section 2 introduces the data, section 3 develops a framework for the validation and calibration of probabilistic predictions, paying particular focus to balancing false positives and false negatives. It then details an application using a Random Forest algorithm. Section 4 presents key results, and section 5 concludes. Additional results are found in the supplementary appendices.

> 2Apart from the food insecurity literature mentioned here, the paper also relates to the work of Celiku and Kraay (2017) who explore predicting conflict outbreaks in the similar context of weighted prediction loss functions.

3
# **2 Data**

## **2.1 Target variable: food crisis outbreaks**

The aim of this paper is to forecast transitions into critical states of food insecurity with enough lead time to take action. We obtain historical data on food insecurity from periodic assessments performed across 1,162 districts in 21 developing countries over the period from 2009 to 2020, obtained from FEWS NET.<sup>3</sup> Food insecurity is measured using the Integrated Phase Classification (IPC) system, an analytical framework that follows evidencebased guidelines to qualitatively classify the severity of food insecurity and prescribe policies to mitigate risk (Hillbruner and Moloney (2012) review the process). The IPC scale distinguishes five phases of food insecurity: (1) minimal, (2) stressed, (3) crisis, (4) emergency and (5) famine. Our predictions are focused on a binary “crisis indicator” defined as IPC ratings of crisis or worse, i.e. phases (3), (4), and (5). This choice is motivated by the fact that the IPC scale recommends a significant shift in policies in order to mitigate, rather than manage the risk of, severe malnutrition outcomes and death once a district enters crisis. In particular, during stressed (2) conditions and below the recommended focus is on risk management, while at crisis levels (3) and above this turns into urgent action to mitigate outcomes.

The FEWS NET data are reported at a sub-national level that follows a combination of administrative boundaries (admin2) and “livelihood zones”.<sup>4</sup> A one-on-one mapping from livelihood zones to administrative districts does not necessarily exist and the former may change over time based on expert opinions. To obtain a consistent time series, we map all the data to standard admin2 districts using a spatial overlay, assigning values to districts by majority coverage. The FEWS NET data are reported at quarterly frequency from 2009 to 2016, and then every four months afterwards. Our sample of food crisis observations used for estimation and cross-validation purposes extends through February 2019. Data for June and October 2019, and February 2020, are considered purely for temporal holdout validation. We develop predictions up to a full year ahead, through February 2020, using covariates that run through February 2019. Some of the district assessments are missing. However, our approach allows us to generate predictions for all districts where covariates are available, even if food crisis outcomes were not.

The IPC rating scale is intended to reflect the actual on-the-ground conditions inclusive of any humanitarian assistance. Since we are interested in predicting food crises themselves and not the impacts of associated humanitarian response, we need to net out the latter. Since 2012, the FEWS NET data marks locations where humanitarian assistance was significant enough to have reduced the IPC phase by one. These markers are used to reconstruct the relevant minimum IPC phases that would have been issued absent humanitarian presence. Figure A.1 in supplementary Appendix A contains the resulting data and Table 1 summarizes.

> 3FEWS NET is a leading food insecurity information provider funded by USAID that regularly publishes analysis and data on current and future food insecurity outcomes.

> 4Admin2 districts are defined by Global Administrative Unit Layers (GAUL); livelihood zones are available from FEWS NET and define sub-national geographic areas of a country, that may cut through administrative borders, in which people are assessed to share similar options for obtaining food and income and have similar access to markets.

4
The summary statistics reveal several important basic insights. Population-weighted average IPC ratings across districts generally are lower than simple averages, indicating that food insecurity is more common in sparsely-populated districts. The minima and maxima highlight that most countries have at some point experienced crisis or worse, but famine declarations (IPC rating of 5) are extremely rare occurrences. Zambia is the only country in our sample that has not experienced food crisis in this period. The number of observations available in each country is listed, with “Crisis+” denoting the number of food crisis observations. “Outbreaks” correspond to the subset of food crisis observations that were preceded by a phase 1 or phase 2 observation. Predicting these transitions into food crisis is the primary goal of this paper. While approximately 16% of observations correspond to food crises, only a small share of these classify as transitions into new outbreaks of food crisis (1,723/6,143 _∼_ 28%). This highlights why the issuance of timely warnings is important: over two-third of classified transitions into crisis are followed by extended periods of crisis, at which point it is too late for prevention.

Table 1: Descriptives of FEWS NET ratings aggregated to admin2, netted of humanitarian assistance effects, used for estimation and cross-validation.

|||**IPC Rati**<br>|**ng**<br>||**Crisis **<br>|**Indicator**<br>||**Observat**<br>|**ions**<br>|**Co**<br>|**verage**<br>|
|---|---|---|---|---|---|---|---|---|---|---|---|
|**Country**|Mean|Pop. Mean|Min|Max|Mean|Pop. Mean|Total|Crisis+|Outbreaks|Districts|Time Periods|
|**Afghanistan**|1.41|1.40|1|4|0.08|0.07|1,224|94|59|34|36|
|**Burkina Faso**|1.08|1.08|1|3|0.00|0.00|1,440|5|5|45|32<sup>_c_</sup>|
|**Chad**|1.45|1.41|1|4|0.10|0.09|2,448|251|129|68|36<br>|
|**Congo, Dem. Rep.**|1.91|2.00|1|4|0.21|0.26|218|46|11|50<sup>_a_</sup>|8<sup>_d_</sup>|
|**Ethiopia**|1.83|1.74|1|4|0.26|0.22|2,664|681|174|74|36|
|**Guatemala**|1.43|1.30|1|3|0.08|0.05|792|62|28|22|36|
|**Haiti**|1.51|1.38|1|3|0.07|0.04|1,512|103|64|42|36|
|**Kenya**|1.46|1.42|1|4|0.08|0.08|2,556|213|60|71|36|
|**Malawi**|1.48|1.48|1|4|0.16|0.16|972|152|87|27|36|
|**Mali**|1.19|1.10|1|4|0.04|0.01|1,800|68|48|50|36|
|**Mauritania**|1.43|1.44|1|4|0.07|0.09|442|32|16|13|34<sup>_c_</sup>|
|**Mozambique**|1.18|1.13|1|3|0.05|0.04|396|21|7|11|36|
|**Niger**|1.36|1.27|1|4|0.07|0.05|2,412|173|94|67|36|
|**Nigeria**|1.78|1.20|1|4|0.20|0.02|3,564|710|231|99|36<br>|
|**Somalia**|2.48|2.44|1|5|0.40|0.37|2,590|1,040|188|74|35<sup>_d_</sup>|
|**South Sudan**|2.13|2.16|1|5|0.33|0.35|2,808|937|232|78|36<br>|
|**Sudan**|1.41|1.34|1|4|0.10|0.07|2,640|265|107|80|33<sup>_d_</sup>|
|**Uganda**|1.10|1.06|1|3|0.02|0.01|2,016|33|12|56<br>|36<br>|
|**Yemen, Rep.**|3.42|3.26|2|4|0.93|0.89|960|892|35|69<sup>_b_</sup>|14<sup>_b,d_</sup>|
|**Zambia**|1.02|1.01|1|2|0.00|0.00|2,304|0|0|72|32<sup>_c_</sup>|
|**Zimbabwe**|1.54|1.44|1|4|0.17|0.13|2,160|365|136|60|36|
|**Total**|1.60|1.41|1|5|0.16|0.10|37,918|6,143|1,723|1,162|116 **months**<sup>_e_</sup>|

The descriptive statistics are calculated using FEWS NET ratings, netted of humanitarian assistance effects, taking admin2 districts as the level of observation. “IPC Rating” refers to the netted data on the original 1-5 scale, while “Crisis Indicator” refers to the binary indicators of phases (3) and above. The data dimensions under “Coverage” refer to the number of admin2 districts in the country, disregard of whether all of these districts cover livelihood zones that receive ratings from FEWS NET, and the number of assessment periods from the first assessment to the last (February 2019). The June/October 2019 and February 2020 assessments are used only for validation and not included in this table. Further guidance on data coverage is as follows: _a_ partially covered, _b_ missing at random, _c_ latest assessment periods missing, _d_ earliest assessment periods missing,<sup>_e_</sup> from first to last assessment: July 2009 - February 2019.

# **2.2 Covariates of food crisis events**

We forecast transitions into food crisis using only readily-observable covariates, available monthly at the admin2 district level prior to outcomes. We deliberately do not use lagged 5
values of the IPC ratings as predictors.<sup>5</sup> This allows us to generate predictions that can be updated monthly based on available data, independently of whether recent IPC ratings are available. We distinguish four key groups of covariates:

# **Structural factors**

Structural factors capture time-invariant vulnerability to food insecurity. We proxy these at the district level with spatial trends, population counts, land area, terrain ruggedness and land use shares (cropland and pasture). In addition, we include fixed effects to capture basic country-specific seasonal variation. Nonlinear prediction models like the Random Forest we deploy can exploit interactions between any (combination) of structural factors and other time-varying covariates to capture heterogeneity.

# **Environmental factors**

We rely on remote sensing data to track environmental factors relating to food production. Food production depends on soil moisture content, which can be monitored through the inputs (rainfall) and outputs (evapo-transpiration) of the water balance equation. We also proxy for food plant health directly using the normalized difference vegetation index (NDVI), which is a commonly-used satellite-imagery-based measure of vegetation coverage (Ross et al., 2009; Brown, 2010).

# **Violent**

Violent conflict is an important factor that can impact food insecurity, by disrupting social systems and blocking access, or may signal other factors that contribute to food insecurity. For example, riots and protests may be the sign of an ailing economy, while violent attacks on civilians by non-state actors can cause displacement of people and disrupt food production and distribution (Br ̈uck and D’Errico, 2019; Maxwell et al., 2020). We use the count of conflict events and the number of fatalities in order to capture events of differing intensity, ranging from protests and riots to lethal attacks.

# **Food price inflation**

Food price inflation can directly cause food insecurity by raising the cost of living for vulnerable households (Baffes et al., 2008; Concei ̧c ̃ao and Mendoza, 2009). Rising prices can also signal whether fiscal space is adequate or tightened, for example due to episodes of conflict (Adam et al., 2008). Over longer time horizons, price uncertainty may lead businesses to postpone investments, see (Koomen et al., 2015; Andr ́ee et al., 2017) for the specific case of agricultural investments. We use monthly sub-national food price inflation estimates developed by Andr ́ee (2020a) to capture these important dynamics.<sup>6</sup>

> 5The impact on the forecasting performance is insignificant but the choice results in larger models. Note that, under suitable conditions, a finite-order moving average approximation of a lower-order autoregressive process remains reasonably accurate. Since innovations can be substituted by exogenous variables by defining a process for lower-level constituents, stable autoregressive processes may also be approximated using a sufficient number of lags of relevant exogenous variables.

> 6The paper assembles sporadically available food price data collected by the World Food Programme, and uses a stochastic imputation that leverages unobserved cross-correlations between intermittent market-level price time series of individual food items to impute price data for up to six staple foods for all markets and months in a given country. These are aggregated into a simple equally-weighted geometric price index and interpolated geographically using inverse distance weighting.

6
## **2.3 Data processing**

Table A.1 summarizes the basic covariates and their sources. To better serve decisionmaking, a more transparent feature set is preferred over a highly processed one. For this reason, we perform only minimal additional data processing. Running maxima of conflict counts and intensity are calculated to capture the worst events that have occurred in the recent past. Second, food price inflation and changes therein are calculated over alternative time windows.<sup>7</sup> After adding these features, spatial averages using 4-nearest-neighbors are calculated for all predictors to capture regional dynamics. The entire feature set is then lagged temporally, thus including temporal lags of spatial lags, to capture dependence on the recent and more distant past. Up to the 12-th lag is included of all time-varying predictors, always ensuring that only information prior to an outcome enters the models. For example, the 4-month ahead predictions use the 4-th to 12-th lag, while the 12-month ahead predictions only use the 12-th lag.

To keep the number of predictors at a modest level, we restrict the maximum linear correlation between any two predictors in the covariate space. In particular, we calculate pair-wise correlations and select predictors that have correlations greater than 0 _._ 75, and, from this subset of highly correlated predictors, drop the one with the highest overall correlation. We repeat this process iteratively until the 0 _._ 75 threshold is no longer breached. The result is a set of processed features that is similar in dimension to the original unprocessed feature set (including temporal lags), while capturing more diverse signals, see Appendix Table A.2.

To assess the benefit of these processing steps, we also evaluate the predictive performance of simply using the original unprocessed features of Table A.1 and their temporal lags.<sup>8</sup> We refer to this as the _“basic”_ feature set, and the more processed feature set as the _“wide”_ feature set. Results show that the _“wide”_ feature set provides important benefits, hence it is the main focus of the paper. Additional detail on the comparison of the _“wide”_ and _“basic”_ feature sets is available in supplementary Appendix B.

# **3 Prediction and Validation Framework**

## **3.1 Loss functions**

We calibrate our predictive model to minimize prediction loss functions that penalize false positives (incorrectly predicting food crises that do not occur) and false negatives (failing to predict food crises that do occur). We allow for a range of relative weights on these two types of errors. To specify these prediction loss functions, the following notation is helpful. Recall that our target variable is a binary variable taking the value 1 if a food crisis (defined

> 7 Specifically: a 3-month running maximum of the 3-month simple moving average of both conflict counts and death intensity, a 6-month running maximum of the 6-month simple moving average of both conflict variables, a 3-month and 6-month difference of the log food price index, and a 3-month, 6-month and 12-month difference of monthly year-to-year food price inflation.

> 8Allowing for pair-wise correlation up to 0.95 to stay as close to the “unprocessed” situation but avoiding possible numerical problems with some of the other, linear, prediction algorithms that were tried. The Random Forest leads to nearly unchanged results if this step were to be dropped as it performs feature selection. Note, however, that variable importance scores may not provide a realistic estimate as two highly correlated variables can both be used to split a decision tree without concrete preference for any one variable.

7
as IPC categories, 3, 4, or 5) is observed, and 0 otherwise. For convenience we sometimes refer to 0 and 1 generically as “class labels” and the observations corresponding to food crises as the “positive class”. Define _Y_<sup>_∗_</sup> as a column vector containing the class labels, stacking all countries/districts/months where an IPC rating is observed. Define _Y_<sup>ˆ</sup><sup>_∗_</sup> as the corresponding column vector of binary predictions, and _P_<sup>ˆ</sup><sup>_∗_</sup> _∈_ [0 _,_ 1] as a column vector of predicted probabilities of the positive class.

The binary outcomes and corresponding predictions are defined for all entries in the data where an IPC rating is observed. However, since our primary focus is on predicting transitions into food crisis, we focus on a subset of observations when evaluating predictions. This subset consists of all country/district/month observations containing an IPC rating, and for which the previous IPC rating is 1 or 2, i.e. not a food crisis. That is, this subset consists of all non-crisis observations that either are followed by another non-crisis observation, or by a transition into food crisis. In this way, we eliminate from the validation set all observations that correspond to ongoing food crises and focus our validation exclusively on predictions of outbreaks of new food crises – before they occur. More specifically, define _Y_ , _Y_ ˆ and _P_ ˆ as the sub-vectors of _Y_<sup>_∗_</sup> , _Y_ ˆ<sup>_∗_</sup> and _P_ ˆ<sup>_∗_</sup> containing the entries that were preceded by non-crises, let I denote a conformable vector of ones, and let w _∈_ [0 _,_ 1] denote a scalar weight. We evaluate predictive performance using these two prediction loss functions: and Note that in _LB_ , the natural log operator ln( _·_ ) is understood to apply element-wise. These loss functions are oriented so that lower values of _LA_ and _LB_ correspond to better predictions. Both loss functions are weighted averages of two terms corresponding to prediction errors made when the true state is 1 versus 0. _LA_ is a weighted average of the false negative rate and false positive rate while _LB_ is a similarly weighted Log Loss function.

The difference between Equation 1 and Equation 2 is that _LB_ simply replaces binary entries used to count the false positives and false negatives, with a continuous penalty that scales with the logarithm of the predicted probability. This means that, when a transition into food crisis occurs, models that fail to predict these transitions with a high probability are penalized exponentially-heavily for predicting values closer to 0. Vice-versa, if the model predicts a high probability of a transition into food crisis and it does not occur, the penalty again increases exponentially, only now for predictions closer to 1. Calibrating a prediction algorithm to minimize _LB_ will thus result in probabilities that are close to 1 (0) when transitions into food crises do (do not) occur. We calibrate our prediction models to minimize _LB_ , and we also report false negative and false positive rates together with _LA_ because of its more intuitive interpretation.<sup>9</sup>

> 9The motivation behind this approach is that _LB_ is a proper scoring rule (Gneiting and Raftery, 2007) that ranks models according to their distance to the optimal model (Andr ́ee, 2020b). Minimizing _LB_ can be understood as maximizing the expected utility of a predictive distribution such that w determines utility trade-off. Propriety encourages the model to make careful predictions and to be honest about the level of

8
In both loss functions, the scalar parameter w governs the relative weight assigned to prediction errors made when the actual outcome is or is not a transition into food crisis. The paper considers three values with a clear policy interpretation. For identical rates,

- w =<sup><u>1</u>failingtoanticipateacrisisishalfascostlyasraisingafalsealarm,</sup> 3<sup>:</sup>

- w =<sup><u>1</u>failingtoanticipateacrisisisascostlyasraisingafalsealarm,</sup> 2<sup>:</sup>

- _•_ w =<sup><u>2</u>failingtoanticipateacrisisistwiceascostlyasraisingafalsealarm.</sup> 3<sup>:</sup>

In our context, this weighting of prediction errors is important since transitions into food crisis are rare but costly. Minimizing unweighted loss would result in a prediction model that emphasizes correct prediction of non-food crisis observations and neglects transitions into food crisis simply because they are rare. As an example, with our data, a model that predicts none of the transitions can score an overall Accuracy on all data of over 95%. In the weighted loss functions, model performance is instead quantified based on a pre-specified policy interpretable parameter rather than the data frequencies.

To calibrate predictions effectively for the different weightings, we follow the strategy proposed by Andr ́ee and Kraay (2020). The idea here is simple. Predicted class probabilities that strongly discriminate between outcomes are more useful for policy purposes than ones that do not. For example, a prediction model that assigns probabilities of 0 _._ 49 to the 0 class and 0 _._ 51 to the 1 class is less useful to discriminate between risks than one that assigns a low (high) probability to the former (latter). Moreover, the weight in the loss function determines whether the preference for prediction bias is upward or downward. _LB_ (w) can often be improved by adjusting both the level of confidence and the general direction of probabilities toward the high-cost outcome. We make use of a simple transformation of the predicted probabilities. The predicted class labels are generated according to: where the predicted probability _P_<sup>ˆ</sup> = _g_ ( _P_<sup> ̄</sup> ; _α, β_ ) is a transformation that depends on two tuning parameters _α_ and _β_ and an input probability _P_<sup> ̄</sup> generated by a model of interest as follows: The parameter _β ∈_ [0 _,_ 1] plays the role of a cutoff probability above which the predicted probability for the positive class should be increased. The parameter _α ∈_ R _>_ 0 controls the confidence of the transformed probabilities, with higher values of _α_ bringing _P_<sup> ̄</sup> closer to

> certainty. In particular, _LB_ encourages confidence in probabilistic predictions but punishes over-confident predictions. This means that when the _LA_ of the optimal model is low, then minimizing _LB_ results in probabilities close to actual outcomes. If, on the other hand, _LA_ of the optimal model is high due to a high degree of randomness in the data, then minimizing _LB_ results in probabilities that do not strongly discriminate between outcomes. Note that when the outcome is fully random, _LB_ is minimized by predicting a probability equal to w. This is easily verified. This shows a straightforward connection between the two loss functions. Drawing class labels randomly with the probability that minimized _LB_ (w) on a random data set results in a ratio between the false positive rate and false negative rate that is equal to their relative weight in _LA_ (w).

9
zero (one) when the input probabilities are below (above) the cutoff value. In this paper we work with a finite number of values _α ∈_ [0 _._ 2 _,_ 2].<sup>10</sup> The optimization and validation is thus summarized as follows. We generate predicted probabilities _P_<sup> ̄</sup> from a statistical model, which may depend on classical tuning parameters. We then generate transformed probabilities _P_<sup>ˆ</sup> using Equation 4 and choose values for _α_ and _β_ that minimize _LB_ . The optimized probability predictions are used to generate class labels using Equation 3 which are finally used to evaluate _LA_ . Note that, if _P_<sup> ̄</sup> is already appropriately balanced and of the preferred level of confidence, then hyper-tuning Equation 4 will lead to minimal re-scaling as the untransformed probabilities are nested by the function. Hence, the strategy can effectively be combined with other techniques for unbalanced learning tasks (including up-sampling).

# **3.2 Cross-validation setup**

We perform 5 repetitions of 10-fold cross-validation.<sup>11</sup> The full sample _Y_<sup>_∗_</sup> consists of 37,198 country/district/month observations containing an IPC rating. The sub-vector of validation cases _Y_ contains 30,948 cases, eliminating all cases in which a food crisis observation is preceded by another food crisis observation. We train the model using the full sample, since a prediction model that uses a larger and more diverse sample of crisis cases achieves lower _LB_ even when it is evaluated only on the transition sample _Y_ . The folds are mutually-exclusive and exhaustive equally-sized partitions of the full allowable validation sample _Y_ , defined as _Ytest,_ 1 _, Ytest,_ 2 _, ...Ytest,k_ =10. These partitions are generated by stratified random sampling to preserve the overall class distribution of _Y_<sup>_∗_</sup> within each fold.<sup>12</sup> The 10 training sets are generated by removing the validation cases from the full sample according to _Ytrain,k_ = _Y_<sup>_∗_</sup> _\ Ytest,k_ .

Given the relatively small and unbalanced data set, we create additional synthetic cases followed by standard up-sampling. In particular, while predictors are defined for each month, _Ytrain,k_ is only defined for the rows in the data for which IPC assessments are available. We create additional training cases by assigning each binary IPC outcome to the previous and following months. The synthetic training examples are added within folds, only to training cases and not validation samples. Standard up-sampling is then applied to the result.<sup>13</sup>

> 10The paper considers only values of _α_ up to 2 although higher values are possible and would result in even stronger transformation of predictions. The reason to investigate only up to a value of 2, implicitly confining our search only to reasonably smooth transformations, is to avoid over-fitting the relatively scarce number of validation samples.

> 11Standard cross-validation can be applied in the context of spatially and temporally dependent observations if the model is flexible enough to over-fit the data, see Bergmeir et al. (2018). If the model over-fits the data strongly, it will score badly in the cross-validation. If the model under-fits the data, and residuals are still correlated, then the independence assumption is invalidated. If the level of mis-specification is mild, and residual dependence is sufficiently weak, then corrections can still be made to the LLN (P ̈otscher and Prucha, 1997; Andr ́ee, 2020b). Note that Random Forest is a method that eagerly fits correlations in a data set of modest dimensions.

> 12Stratified sampling ensures folds always contain observations in both classes. Note that Equations 1 and 2 average two components, hence consistent estimation requires consistent estimation of two lower level components. Thus, the validation strategy requires a sufficient number of validation cases in both classes, but the exact class distribution in the validation sample does not matter for our loss criteria as w is pre-defined.

> 13This is important, adding synthetic training cases based on a 1 month window around the test cases would invalidate the weak dependence assumption, needed for consistent validation, that is imposed at the 3- to 4-month interval. The approximate size of each resulting training set is _∼_ 105 _,_ 428 samples.

10
# **3.3 Application using Random Forest**

The prediction and validation framework is applicable to any binary classification algorithm that generates predicted probabilities for the positive class. In this paper we use a Random Forest classifier.<sup>14</sup> We focus on this particular approach in light of the additional results provided in supplementary Appendix B which show that in this same data set, the Random Forest offers a substantial improvement in predictive performance over simple linear models, but that little further is gained by moving to more complex learners such as neural networks.

Implementing the algorithm requires specifying several tuning parameters that control the structure of the forest or its trees. We select these to optimize predictive performance using the cross-validation procedures described above. This is implemented by searching over a practical grid of values.

First, since we require predicted probabilities, we implement a probability forest in which trees are grown as regression trees following Malley et al. (2012). An important tuning parameter is minimum node size that regulates tree depth by setting the minimum number of cases at the terminal nodes of the decision trees. We use the original value of 1, as well as the default value of 10 in the implementation of Wright and Ziegler (2017).<sup>15</sup> The out-of-sample performance is heavily dependent on the predictive power of individual trees and the correlation between them.<sup>16</sup> The relationship between the two is mainly one of trade-offs and can to a certain degree be regulated by optimizing over the number of variables that are used to grow individual decision trees.<sup>17</sup> This number is controlled by _mtry_ , and its default value is ~~�~~ ( _D_ ) where _D_ is the number of predictors in the data set. Performing a grid search for _mtry_ is computationally challenging and in this paper is made feasible by focusing on particular combinations of tuning-parameters.<sup>18</sup> In particular, the grid search is performed only when the minimum node size is 10, and makes use of a

> 14The algorithm was introduced by Breiman (2001) as a generalization of the tree bagging by combining it with the random subspace method of Ho (1998). It has quickly become a popular non-parametric classification tool due to its ability to generate good results using standard tuning parameter values and its robustness to noise. It constructs prediction rules without imposing strong prior assumptions on the functional form of the relationship between predictors and the target variable. Good review articles include those of Biau (2012); Biau and Scornet (2016). Random Forest generates results that are accurate while relatively straightforward to interpret (Banerjee et al., 2012; Petkovic et al., 2018).

> 15Breiman’s original algorithm grows trees to purity, that is, until each terminal node contains only observations from one class. When nodes are pure, the probability estimate at each terminal node is thus always either 0 or 1. This means that, apart from a low correlation between trees, a large number of them is needed to obtain precise estimates of probabilities by the forest. Increasing minimum node size allows the frequency of class occurrence at the terminal nodes itself to be a reasonable probability estimate.

> 16The generalization error converges _almost surely_ to a limit as the number of trees in the forest grows. We use 500 trees, corresponding to the default value in the implementation by Wright and Ziegler (2017). The magnitude of expected out-of-sample error in the limit in turn depends on the predictive strength of the individual trees and the correlation between them. See also (Scornet, 2017).

> 17The predictive strength of individual trees can to a degree be improved by allowing more variables to enter individual base learners. However, while larger values for _mtry_ may result in stronger base learners, it also leads to a higher correlation between them. In particular, as _mtry → D_ , the probability that two base trees contain the same variables approaches 1. If many individual trees contain a highly similar set of variables, then their predictions become more correlated. Smaller values for _mtry_ may instead increase diversity of base learners, at the cost of lower individual predictive power.

> 18Larger values of _mtry_ significantly increase computational cost as it becomes more complex to process the individual trees. A grid search for _mtry_ is more manageable in combination with tuning values that lead to simplifications elsewhere in the calculations; higher minimum node size that leads to faster individual tree-building, and randomized splitting does not require calculating Gini coefficients.

11
faster splitting rule introduced by Geurts et al. (2006).<sup>19</sup> Table 2 lists the full combination of tuning parameters. Overall, tuning the parameters of the probability transformation in Equation 4 ( _α, β_ ), had by far the biggest impact on performance with respect to our balanced evaluation criterion.

Table 2: Random Forest tuning parameter combinations.

|**splitrule**|**mtry**|**min.node.size**|**_α_ (probability scaling**|**)**<br>**_β_ (probability constant)**|
|---|---|---|---|---|
|Gini|_~~√~~_<br>_D_|1, 10|(1:10)/|5<br>(1:100)/100|
|ExtraTrees|_√_<br>_D_|1, 10|(1:10)/|5<br>(1:100)/100|
|ExtraTrees|(0_._5_,_0_._75_,_1_._25_,_1_._5)_×_<br>_√_<br>_D_|10|(1:10)/|5<br>(1:100)/100|

The column names correspond to the tuning-parameters considered in the tuning grid. Row entries list combinations of values that result in unique model specifications. Thus (bottom row), when **splitrule** equals ExtraTrees and **min.node.size** equals 10, a range of different values for **mtry** is considered. Moreover, for each possible combination of **splitrule** , **min.node.size** and **mtry** , 10 _×_ 100 unique combinations of ( _α, β_ ) are tried.

# **4 Results**

In this section we report on the prediction performance of our main model. Section 4.1 investigates the prediction of food crisis outbreaks sub-nationally, using the prediction and validation framework of the previous section. Section 4.2 investigates forecast results aggregated to a country level, again relying on cross-validation procedures. We also validated the models in a temporal holdout exercise. Overall, we find that this additional validation exercise corroborates the main findings presented below, the additional results are contained in supplementary Appendix B, section B.2. Finally, section 4.3 describes procedures to interpret the relative importance of different factors for model predictions.

## **4.1 Validation of district-level predictions**

To set a performance baseline, we begin by summarizing the predictive performance of a simple binary crisis indicator derived from the existing future outlooks produced by FEWS NET. These follow the identical definition as our target variable, only now applied to the future outlooks.<sup>20</sup> These expert-based outlooks are closely watched by policy makers, and so their near-term and medium-term binary equivalents provide a natural benchmark with

> 19The original approach splits trees based on a measure of node purity. This increases the probability that the same variables ends up at the majority of root nodes when _mtry_ increases, leading to high similarity between trees. Countering the correlation through randomization can produce stronger and more diverse trees that improve forest accuracy. This paper implements randomization of splits using the _extraTrees_ algorithm introduced by (Geurts et al., 2006). This selects cut-points at random, i.e., independently of the target variable. Key advantages are potentially increased forest accuracy due to lower tree correlation and faster runtime which makes tuning _mtry_ more practical and more interesting.

> 20In addition to its categorical ratings of actual food insecurity conditions, FEWS NET also reports a “near-term outlook” and a “medium-term outlook”. These outlooks represent preliminary assessments of the most likely food security ratings one and two reporting cycles ahead. The near-term outlook corresponds to either 3 or 4 months ahead while the medium-term outlook corresponds to either 6 or 8 months ahead, depending on the frequency of the reporting cycle which increased to 4 months in 2016. Recall that these outlooks may also follow livelihood zones, in which case we aggregated them to the district level. In an identical treatment to the outcomes, we also net the outlooks from the projected humanitarian assistance effects, and thus validate the binarized indicators purely on their ability to signal the need for action, disregard of whether that action was already anticipated. This is in line with the use case explored for model-based forecasts.

12
which to compare the model-based forecasts. Because FEWS NET data do not assign a probability to future outcomes, we only focus on the _LA_ loss function of Equation 1. Table 3 contains the results.

The most striking feature of Table 3 is the extreme imbalance between false alarms – an FPR of around 2% – and failure to predict – an FNR of over 74%. This clearly indicates that the historical performance baseline is characterized by a strong aversion to predicting transitions into food crises that subsequently do not occur. In the context of our evaluation framework, this can be interpreted as evidence of a very high implicit weight on false positives and a low weight on false negatives, i.e. a low value of w.<sup>21</sup> The low false alarm rate does not mean that those warnings that were raised, did also most likely predict a transition correctly. To the contrary, only 35% of the projected transitions into crisis were followed by actual transitions. Comparing the frequency of transitions into crisis in both the historical assessments and outlooks also highlights the conservative nature. Although transitions into crisis account for 5 _._ 3% of the historical assessment sample, only 3 _._ 2% of the binarized outlooks projected a transition, see Table B.1 in supplementary Appendix B.

Table 3: Historical performance baseline calculated from binarized FEWS NET outlooks.

||**Error **|**Types**||**_LA_**||
|---|---|---|---|---|---|
||**FNR**|**FPR**|**w=1/3**|**w=1/2**|**w=2/3**|
|**Near-term**|74.3%|1.4%|25.7%|37.8%|50.0%|
|**Medium-term**|79.4%|2.2%|27.9%|40.8%|53.7%|

**FNR** and **FPR** of binarized historical FEWS NET outlooks in the near-term (3-4 months, validated against outcomes one reporting cycle later) and medium-term (6-8 months, validated against outcomes two reporting cycles later) and three weighted averages ( **_LA_** ) with weights **w** and (1- **w** ) for prediction errors in the positive (negative) class. Both the binary outcomes and outlooks are netted of humanitarian assistance effects. The validation was performed at the admin2 level, and only for observations that were preceded by non-crisis levels of food insecurity. This specifically describes prediction performance for new outbreaks of food crisis before they occur. The validation sample spans all available projection-outcome pairs from July 2009 - February 2019.

Table 4: Cross-validated performance of predictions from the wide Random Forest.

||||**Error **|**Type**||||**Bal**|**anced M**|**etrics**|||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|||**FNR**|||**FPR**|||**_LA_**|||**_LB_**||
|**w**|**1/3**|**1/2**|**2/3**|**1/3**|**1/2**|**2/3**|**1/3**|**1/2**|**2/3**|**1/3**|**1/2**|**2/3**|
|**h=4**|19.7%|16.5%|16.0%|6.8%|8.2%|8.4%|11.1%|12.3%|13.5%|0.25|0.29|0.31|
|**h=8**|19.1%|16.6%|16.6%|7.3%|8.5%|8.5%|11.2%|12.5%|13.9%|0.27|0.30|0.33|
|**h=12**|20.9%|20.9%|9.9%|10.0%|10.0%|19.0%|13.6%|15.5%|13.0%|0.30|0.34|0.35|

The first six columns report false negative rates and false positive rates for models optimized for the indicated value of **w** . The next six columns report weighted averages and weighted Log Loss values ( **_LB_** ) with weights **w** and (1- **w** ) for prediction errors in the positive (negative) class. The statistics have been cross-validated for **h** =4-, 8- and 12-month ahead forecasts of crisis outbreaks. The validation was performed at the admin2 level, and only for observations that were preceded by non-crisis levels of food insecurity. This specifically describes prediction performance for new outbreaks of food crisis before they occur. The validation sample spans from July 2009 - February 2019.

> 21Recall that unbiased Log Loss sets w to the frequency of the positive class in the validation sample, 5 _._ 3% on average in random draws, hence an unbiased model that assumes both the false positive rate and false negative rate carry the same cost, would assume a value of around w _∼_ 0 _._ 05. The binarized outlooks produce even fewer alarms, at a rate of 3 _._ 2%, meaning that implicitly w _<_ 0 _._ 05. At, w= 0 _._ 03 the rate of false alarms is weighted 30 times as much as the rate with which anticipating crises fails in a weighted cost function like _LA_ .

13
The statistical model-based predictions we develop have the potential to improve on this baseline in several ways. We explore this using Table 4, which provides cross-validation results for 4-, 8- and 12-month ahead forecasts. Note that the 4- and 8-month forecast horizons are roughly comparable to the near- and medium-term outlooks considered by FEWS NET.

First, when we set a conservative value of w=<sup><u>1</u></sup> 3<sup>toimposeaheavierpenaltyonfalse</sup> alarms, the FPR increases slightly relative to the performance baseline produced with binarized FEWS NET outlooks, to around 7% for the 4- and 8-month forecast horizon. However, this comes with a very large reduction in the FNR to below 20% which is substantially down from the baseline rates of 74% and 79%. Taken together, the slightly higher FPR and much lower FNR results in an improvement in _LA_ from the baseline value of around 26% using binarized FEWS NET outlooks, to around 11% for the model predictions.

Second, the model-based approach allow us to fine-tune predictions for different use cases corresponding to different weights in the prediction loss function. As we have discussed, the severe costs of food crises suggest that for some policy purposes, a greater penalty should be imposed on failure to predict crises, i.e. that a higher value of w would be warranted. The FEWS NET outlooks offer just one fixed set of forward-looking ratings, and increasing the weight on FNR in the prediction loss function in Table 3 sharply increases _LA_ , indicating a sharp decline in predictive performance with regard to the less conservative forecasting objectives. For example, for the medium-term outlook, _LA_ increases from 28% to 54% when w increases from<sup><u>1</u></sup> 3<sup>to</sup><sup><u>2</u></sup> 3<sup>.Incontrast,themodelpredictionsmaintaincomparablepredictive</sup> performance across the range of weights, as evidenced by minimal changes in the weighted performance metrics.

Third, models can learn different predictive associations depending on the forecast horizon and tailor predictions to the lead-time that is required. The complexity associated with unlocking funds and investing in prevention implies that near-term forecasts have lower utility than medium-term forecasts when equally accurate and so it would be helpful to predict further into the future with similar accuracy. The FEWS NET outlooks offer projections that run roughly 8 months into the future which, when binarized, score an _LA_ of 28% for the conservative weight of w=<sup><u>1</u></sup> 3<sup>.We assessed the ability to predict 12 months ahead,</sup> and find that the model maintains an _LA_ below 14%, still roughly half of the medium-term performance baseline.

Fourth, models have the ability to predict the probability of an event which means they can rank districts according to risk. _LB_ measures how well the predicted probabilities align with the outcomes and rewards predictions that trend in the right direction. The minimum value of _LB_ depends on the degree of noise in the target variable but it is well-known that an uninformative model, that predicts only the correct average probability, attains a Log Loss of _._ 69.<sup>22</sup> Hence, any value below _._ 69 for any value of w implies that the model not only

> 22As a reference, it can be shown that in a system, where you would know all the possible information about its future state other than some unknown internal level of randomness, so that, given that all model parameters are at their correct values, an outcome can still be different than predicted. Say for example that in this system the best one can do is predict 0.1 when the outcome is 0, and .9 when the outcome is 1, and that for one out of ten predictions the outcome falls into the other category. Then, Log Loss attains a minimum of approximately _._ 33.

14
learned to predict the preferred average risk level but that, on average, the probabilities also trend in the right direction. The _LB_ values thus indicate that the models predict good probabilities.

Note that standard tuning parameters do not explicitly re-distribute predictions toward a high-cost outcome. The ability to tune predictions for the differential in costs instead relies on the transformation of the predicted probabilities controlled by tuning parameters ( _α, β_ ).<sup>23</sup> For example, the untransformed “Vanilla” model at _h_ = 4, tuning only over default parameters – and not ( _α, β_ ) – benefits from up-sampling but has no explicit flexibility to deal with the unbalanced prediction problem and reaches an _LB_ �w = ( 3<sup><u>1</u></sup><sup>_,_</sup><sup><u>1</u></sup> 2<sup>_,_</sup><sup><u>2</u></sup> 3<sup>)</sup> � of ( _._ 31 _, ._ 41 _, ._ 51). The associated values of _LA_ are (15 _._ 3% _,_ 21 _._ 9% _,_ 28 _._ 5%). Finally, it is also possible to tune ( _α, β_ ) to obtain an extremely conservative model. For example, when minimizing FNR under the constraint that FPR may not increase over the 1 _._ 4% set by the performance baseline, the model produces an FNR of 49% and an _LA_ of 17 _._ 3% at w=<sup><u>1</u></sup> 3<sup>,downfromthe</sup> baseline value of 26%.

Table 5: Country-specific validation results of binarized FEWS NET outlooks and predictions from the wide Random Forest.

||**Bin**|**arized **|**Outlo**|**oks**|**R**|**andom**|**Fores**|**t**|
|---|---|---|---|---|---|---|---|---|
||**Near-**|**term**|**Med.**|**-term**|**h=**|**4**|**h**|**=8**|
|**w**|**1/3**|**2/3**|**1/3**|**2/3**|**1/3**|**2/3**|**1/3**|**2/3**|
|**Afghanistan**|27%|53%|24%|46%|17%|28%|17%|27%|
|**Chad**|30%|59%|33%|63%|7%|9%|7%|8%|
|**Ethiopia**|30%|57%|29%|55%|20%|19%|19%|18%|
|**Haiti**|25%|50%|31%|59%|27%|49%|28%|49%|
|**Kenya**|25%|47%|26%|50%|16%|26%|20%|35%|
|**Malawi**|27%|52%|30%|58%|11%|14%|11%|13%|
|**Mali**|24%|47%|28%|55%|11%|16%|8%|14%|
|**Niger**|22%|41%|32%|60%|9%|14%|10%|16%|
|**Nigeria**|31%|62%|31%|60%|5%|4%|5%|4%|
|**Somalia**|25%|47%|33%|59%|30%|21%|33%|21%|
|**South Sudan**|24%|41%|28%|50%|17%|13%|17%|13%|
|**Sudan**|25%|50%|30%|60%|16%|22%|15%|21%|
|**Zimbabwe**|27%|53%|23%|44%|11%|11%|10%|10%|
|**Balanced average**|26%|51%|29%|55%|15%|19%|15%|19%|
|**Burkina Faso**|0%|0%|33%|67%|28%|45%|33%|67%|
|**Congo, Dem. Rep.**|34%|67%|33%|67%|19%|28%|17%|21%|
|**Guatemala**|30%|59%|30%|59%|23%|41%|23%|41%|
|**Mauritania**|35%|67%|35%|67%|22%|40%|23%|40%|
|**Mozambique**|29%|57%|24%|48%|21%|31%|26%|44%|
|**Uganda**|33%|67%|34%|67%|21%|37%|21%|36%|
|**Yemen, Rep.**|6%|9%|24%|16%|36%|20%|28%|15%|

The first four columns report weighted averages of false negative rates and false positive rates ( _LA_ ) for binarized FEWS NET outlooks in the near-term (3-4 months, validated against outcomes one reporting cycle later) and mediumterm (6-8 months, validated against outcomes two reporting cycles later), for the indicated value of **w** . Both the binary outcomes and outlooks are netted of humanitarian assistance effects. The next four columns report crossvalidated statistics for the wide Random Forest optimized for the indicated value of **w** , for **h** =4- and 8-month ahead forecasts of crisis outbreaks. The validation was performed at the admin2 level, and only for observations that were preceded by non-crisis levels of food insecurity. This specifically describes prediction performance for new outbreaks of food crisis before they occur. Statistics for countries with approximately fewer than 50 historical transitions into crisis are in the bottom part. Zambia is not included in the validation sample due to an absence of positive class values. The validation sample spans from July 2009 - February 2019.

23 <u>1</u> For w= 2<sup>andw=</sup><sup><u>2</u></sup> 3<sup>wefind</sup><sup>_α_=2,ourmaximumconsideredvalue,and</sup><sup>_β_closeto0,indicatingthat</sup> nearly the entire range of predicted probabilities are skewed upward. Recall that the natural frequency of the data favors a w value around 0 _._ 05 and so our predictions need a strong transformation to do well on a loss function that penalizes FNR heavily. Future work with more validation samples could consider higher values for _α_ . For now, we simply note that the results are already extremely competitive for high values of w.

15
It is also interesting to analyze how the error rates vary by country. Table 5 provides a country-level comparison between the previous baseline results and cross-validated model predictions. Recall from Table 1 that the number of historical transitions is low in some countries. This means that _LA_ and _LB_ are easily influenced by a few errors. Countries with approximately 50 or more outbreaks are indicated in the top half of the table. The _LA_ values in these countries vary from 23% in Zimbabwe to 33% in Chad and Somalia at w=<sup><u>1</u></sup> 3<sup>.</sup> The conservative bias in the performance baseline is a feature that persists at the individual country level. When w=<sup><u>2</u></sup> 3<sup>,thebalancederrorratesinmultiplecountriesreachover50%,</sup> in some countries reaching around 60%. These validation results could in expectation be beaten by randomly guessing at the desired frequency.<sup>24</sup> Overall, model-based predictions improve a simple cross-country average of balanced error rates (w=<sup><u>1</u></sup> 3<sup>)from26%inthe</sup> near-term and 29% in the medium-term, to respectively 15% and 19%.

# **4.2 Validation of national-level predictions**

Our modeling approach thus far has focused on generating predictions of food crisis at the sub-national level, focusing particularly on calibrating the predictions to minimize the prediction loss function with varying weights of prediction errors in the positive class. Since finance is often unlocked at a country level, we also explore the properties of the predictions when they are aggregated to form country-level predictions. Specifically, for each country in our sample, we construct the share of the country’s population living in districts that are in food crisis, as measured by an IPC rating of 3, 4 or 5. We then investigate how well the country-level aggregate of our district-level predictions forecasts this aggregate country-level food crisis indicator. This exercise is of interest because scarce global humanitarian resources typically are prioritized between regions of the world and countries first, while the allocations to communities and beneficiaries at sub-national level occurs with a separate targeting process. For example, the World Bank’s base allocation is predetermined at country-level. Having a good forecast of country-level food crisis affected populations can therefore help move resources to the appropriate countries, potentially early enough to finance preventive measures.<sup>25</sup> We construct the country-level aggregate food crisis-affected population prediction as a weighted average of the district-level probabilities in the country, with weights equal to populations living in each district, i.e.

where _i_ , _j_ , and _t_ index districts, countries and time periods and pop _ijt_ represents population. Since the scale of the aggregated country-level predictions depends on the bias introduced

> 24Recall that the uninformative value of _LA_ (w) equals 50% when labels are generated randomly with frequency w. _LA_ (w) will only go above 50% if w (preference for error types) moves in the opposite direction of the model’s bias. For example, it will reach 100% if w=1 while the frequency of positive predictions is 0 (and all the class labels are thus in the negative class, e.g. FPR equals 0 and FNR equals 100%, the latter being fully weighted in _LA_ ).

> 25For an alternative approach to modeling country-level food insecurity over long horizons, up to three years, see Wang et al. (2020) who utilize panel vector auto-regression techniques.

16
by the chosen value of w, the scale of the prediction result may differ from the scale of the actual outcomes. To adjust for this simple bias, we regress the country-level result on the available country-level historical outcomes and use the fitted values as final forecasts.<sup>26</sup> We validate the forecasts by calculating Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE), reported in Table 6.<sup>27</sup> Table 6: Country-specific validation results of crisis-affected population forecasts from binarized FEWS NET outlooks and predictions from the wide Random Forest.

||**B**<br>**Near-**|**inarized **<br>**term**|**Outlooks**<br>**Medium**|**-term**|**h=**|**Random **<br>**4**|**Forest**<br>**h=**|**8**|
|---|---|---|---|---|---|---|---|---|
|**Country**|**RMSE**|**MAE**|**RMSE**|**MAE**|**RMSE**|**MAE**|**RMSE**|**MAE**|
|**Afghanistan**|7.9|3.9|15.5|9.8|5.4|3.2|5.9|3.9|
|**Burkina Faso**|1.9|0.4|2.7|0.8|0.8|0.3|1.1|0.5|
|**Chad**|5.2|2.2|13.2|9.0|6.0|4.5|6.4|4.8|
|**Congo, Dem. Rep.**|5.6|2.5|3.2|1.5|16.9|12.5|18.9|14.6|
|**Ethiopia**|3.8|2.4|16.5|12.8|8.8|7.4|9.5|7.9|
|**Guatemala**|1.1|0.4|6.3|3.5|3.9|3.0|4.0|3.3|
|**Haiti**|3.4|2.1|6.6|4.7|3.8|3.3|4.8|4.4|
|**Kenya**|5.0|2.7|4.2|2.5|6.2|5.2|7.2|6.6|
|**Malawi**|11.6|4.9|31.5|20.7|10.0|7.4|11.1|9.4|
|**Mali**|3.4|1.6|5.4|2.8|1.9|1.1|2.1|1.2|
|**Mauritania**|9.0|3.7|15.5|7.2|10.5|7.7|11.9|9.7|
|**Mozambique**|2.1|0.4|8.0|2.8|4.3|3.5|5.3|4.3|
|**Niger**|10.2|4.1|15.0|6.5|3.8|2.7|4.3|3.4|
|**Nigeria**|0.2|0.0|1.9|1.1|1.1|1.0|1.0|0.9|
|**Somalia**|18.7|8.6|26.5|16.2|21.5|18.2|22.2|18.8|
|**South Sudan**|14.4|8.6|17.4|14.5|11.8|10.7|11.9|10.8|
|**Sudan**|4.8|2.1|9.3|6.5|4.1|3.5|4.3|3.7|
|**Uganda**|0.0|0.0|1.0|0.5|0.9|0.7|1.2|1.0|
|**Yemen, Rep.**|12.4|4.8|16.1|7.1|14.5|8.5|14.3|8.4|
|**Zambia**|0.5|0.1|0.5|0.1|0.0|0.0|0.0|0.0|
|**Zimbabwe**|6.4|3.1|20.6|11.3|7.9|6.2|9.3|7.9|
|**Balanced average**|6.4|2.9|11.8|7.1|7.2|5.5|7.8|6.3|
|**Balanced average**<sup>_∗_</sup>|5.6|2.7|11.9|7.3|5.6|4.4|6.2|5.2|

The first four columns report Root Mean Squared Error and Mean Absolute Error for forecasts of country-level populations in crisis-affected districts, constructed from near-term and medium-term binarized FEWS NET outlooks. Both the binary outcomes and outlooks are netted of humanitarian assistance effects. The second set of four columns report cross-validated statistics of for the wide Random Forest, for 4- and 8-month-ahead forecast horizons.<sup>_∗_</sup> Zambia, Uganda, the Republic of Yemen, Somalia and the Democratic Republic of Congo are excluded for reasons described in the text. The validation sample spans from July 2009 - February 2019.

The results suggest that aggregated district-level predictions are reasonable predictors of country-level outcomes. At both 4- and 8-month forecast horizons, the model produces error rates that are broadly comparable to those obtained with binarized near-term outlooks.<sup>28</sup>

> 26A simple equation that involves only one scalar, e.g. of the form _Y_ = _βX_ + _ε_ . The district-level predictions used here, _P_<sup>ˆ</sup> _ijt_ , are generated setting w=<sup><u>1</u></sup> 3<sup>inthepredictionlossfunctionandareoptimizedas</sup> before to minimize loss at the district level.

> 27Specifically, we generate holdout predictions for all observations in the data set under optimized hyperparameters, and using 5 repetitions of 10-fold cross validation, as described in the previous section. It is important to note that the model hyper-parameters have been optimized to forecast crisis outbreaks, not total crisis-populations. Future efforts could test possible forecast gains when designing a prediction model specifically for the latter purpose.

> 28The balanced average of the model is heavily impacted by Somalia and the Democratic Republic of Congo. Inspection of the predictions in Somalia revealed that a considerable share of the errors was due to over-predicting in the period between 2014-2017 when outcomes briefly hit historical lows of 0%, and under-predicting when the outcomes hit 100%. Generally, we found that the modeled results were actually good leading indicators. For example, in a holdout (June and October 2019), the binarized near- and medium-term outlooks from FEWS NET had an absolute error of 13 _._ 4 and 47 _._ 0 points respectively, while the model errors were 1 _._ 2 and 2 _._ 0 points on the 4 and 8 month horizons. In Democratic Republic of Congo,

17
Importantly, the error of model increases only slightly as we move to the 8-month forecast horizon, while errors made with binarized outlooks nearly double when moving from the near-term to the medium-term horizon. This means that the model-based predictions are better suited for forecasting further into the future, which may help increase lead time for preventive action.

# **4.3 Model interpretability**

The previous sections showed that the Random Forest model provides good predictive capabilities. While predictive performance is important, the modeling structure of the transformed classifier may be difficult to interpret. Out-of-the-box diagnostics may be used to help determine which variables are important in the forest, but do not inform about the signs and magnitude of (local) relationships. Parametric models are more transparent in that regard, but while it can be argued that these models are more interpretable, they might not be comprehensible given the number of predictors involved. Moreover, standard methods do not account for effects induced by the transformation of probabilities. To overcome these issues, this section develops a prediction decomposition strategy that relies on assessing the change in prediction values when the model is confronted with observed values compared to reference input data. This provides comprehensible interpretation at an observational level and takes the following algorithmic strategy.

1. Partition the spatio-temporally covariates into a fully exhaustive set of non-overlapping groups. We use the groups defined in section 2.2 but split rainfall variables from the other environmental factors.<sup>29</sup>

2. Construct a country data set by leaving all time-invariant district-level data at local values and setting all spatio-temporally varying covariates at a reference value. We use country means.<sup>30</sup> Make reference predictions with this data set.

3. For each group of covariates defined in step 1, set the levels of the data to the observed values while keeping all the other covariates at reference values as defined in the previous step, then generate a new set of predictions.

4. For each set of predictions generated in the previous step, subtract the reference predictions generated in step 2. Then, from this prediction difference, calculate the lowest value in each district across all time periods and subtract these.

5. Standardize the prediction differences generated in the previous step to [0,1] by dividing them by their sum, then multiply by probabilities predicted when all data are set to observed levels.

> only 8 assessment cycles are available for cross-validation and errors are heavily driven by the first two periods. Again, good performance was established in a holdout, binarized outlooks scoring respectively 9 _._ 3 and 16 _._ 3 points while the model-based predictions missed the outcome by only 1 _._ 6 and 3 _._ 2 points. Finally, few assessments are available for the Republic of Yemen which has stayed near 100% crisis in all but the first period, no difference was ever recorded between near-term preliminary assessments and final outcomes in Uganda, and Zambia never experienced crisis. When focusing on the remaining 16 countries for validation, RMSE in the near-term was identical for model-based predictions and binarized outlooks, while on the 8-month target the model reduced RMSE by 48% increasing only 0 _._ 6 points compared to the binarized near-term outlook.

> 29“Markets” includes all features calculated from food prices and “Conflict” all features derived from the conflict data. The Environmental Factors are divided in two groups: “Rainfall” and “Agricultural Stress”, the latter combining all the features derived from NDVI and evapo-transpiration.

> 30It would also be possible to construct specific alternatives with which to compare observed values such as conflict counts of 0 rather than a non-zero average.

18
<!-- Start of picture text --> Somalia South Sudan<br>DoQosc ©~ + G @ AgricuMarketsRainfa l ltural Stress @2oc S= 1B@® MarketsRainfallAgricultural Stress © ef@ °<br>zsSc3a2 oSSa | ©@ ConflictHistorical percentages@ (FEWS) eoRMSE: 7.345 MAE:eo 6.41 e S&©a> ©©So6 "| @|B ConflictHistorical percentages (FEWS) RMSE: 3.253 MAE: 2.838<br>xe)38os32~<br>sS Sog SN<br>2 g<br>Bio fio<br>Jan 2008 Jan 2010 Jan 2012 Jan 2014 Jan 2016 Jan 2018 Jan 2008 Jan 2010 Jan 2012 Jan 2014 Jan 2016 Jan 2018<br>Dates through May 2019 Dates through May 2019<br><!-- End of picture text -->
in advance opens important opportunities to mitigate and avoid the worst. In this paper, we have explored the benefits of using formal statistical models to forecast food crises, using sub-national data provided by FEWS NET for 21 countries since 2009.

As a baseline, it is instructive to compare model-based forecasts with existing early warning systems. In our validation framework, these historical outlooks turn out as quite conservative, with extremely low false positive rates and relatively few warnings when compared with the actual incidence of food crisis outbreaks. As a consequence, existing outlooks have high false negative rates, i.e. many food crisis outbreaks were not anticipated. In addition, the overall forecast performance of existing outlooks deteriorates with longer forecast horizons.

Using standard cross- and temporal holdout validation methods, we have shown that relatively simple statistical models provide good predictive performance up to 12 months in advance. We used verifiable monthly data capturing environmental factors, conflict, and food price shocks as key predictors for this. We find that the model-based forecasts can deliver much lower false negative rates at the cost of only modest increases in false positive rates. Moreover, our metrics of forecast performance are fairly stable as the forecast horizon increases from 4 to 12 months, suggesting good predictive performance over longer forecast horizons.

A key benefit of statistical forecasts is that they can be optimized relative to an explicit trade-off between false positives and false negatives. This trade-off may be different for different policy purposes. When the costs of full-blown food crises are large and preventive measures have high returns, a forecasting framework with a high tolerance for false positives might be appropriate. Conversely, when resources for prevention are scarce, a greater tolerance for false negatives is more suitable. Our validation framework made this trade-off explicit, by tuning the parameters of our models to minimize prediction loss functions that balance false positives and false negatives. Depending on the use case, calibrated predictions can be generated that strike the appropriate balance between false positives and false negatives dictated by the application at hand. We acknowledge, however, that putting this principle into practice is a difficult task. While specifying a loss function that trades off false positives and false negatives is mathematically straightforward, setting the appropriate weights requires careful deliberation of the human and economic implications of competing errors, both of which are subject to considerable uncertainty.

Combined, the results show that statistical models can provide new and attractive forecasting capabilities in various ways, including an improved ability to detect new crisis outbreaks for a specified tolerance for raising (false) alarms; calibrating error types to take the costs and benefits of specific interventions into account; increasing lead time for preventive action by forecasting further into the future, and updating predictions continuously using readily-available, verifiable and open high-frequency data. Realizing these benefits will require close coordination between statistical modeling work and policy dialog, to ensure that the design of warning systems is tailored to the policies they are intended to inform.

20