faster splitting rule introduced by Geurts et al. (2006).<sup>19</sup> Table 2 lists the full combination of tuning parameters. Overall, tuning the parameters of the probability transformation in Equation 4 ( _α, β_ ), had by far the biggest impact on performance with respect to our balanced evaluation criterion.

Table 2: Random Forest tuning parameter combinations.

|**splitrule**|**mtry**|**min.node.size**|**_α_ (probability scaling**|**)**<br>**_β_ (probability constant)**|
|---|---|---|---|---|
|Gini|_~~√~~_<br>_D_|1, 10|(1:10)/|5<br>(1:100)/100|
|ExtraTrees|_√_<br>_D_|1, 10|(1:10)/|5<br>(1:100)/100|
|ExtraTrees|(0_._5_,_0_._75_,_1_._25_,_1_._5)_×_<br>_√_<br>_D_|10|(1:10)/|5<br>(1:100)/100|

The column names correspond to the tuning-parameters considered in the tuning grid. Row entries list combinations of values that result in unique model specifications. Thus (bottom row), when **splitrule** equals ExtraTrees and **min.node.size** equals 10, a range of different values for **mtry** is considered. Moreover, for each possible combination of **splitrule** , **min.node.size** and **mtry** , 10 _×_ 100 unique combinations of ( _α, β_ ) are tried.

# **4 Results**

In this section we report on the prediction performance of our main model. Section 4.1 investigates the prediction of food crisis outbreaks sub-nationally, using the prediction and validation framework of the previous section. Section 4.2 investigates forecast results aggregated to a country level, again relying on cross-validation procedures. We also validated the models in a temporal holdout exercise. Overall, we find that this additional validation exercise corroborates the main findings presented below, the additional results are contained in supplementary Appendix B, section B.2. Finally, section 4.3 describes procedures to interpret the relative importance of different factors for model predictions.

## **4.1 Validation of district-level predictions**

To set a performance baseline, we begin by summarizing the predictive performance of a simple binary crisis indicator derived from the existing future outlooks produced by FEWS NET. These follow the identical definition as our target variable, only now applied to the future outlooks.<sup>20</sup> These expert-based outlooks are closely watched by policy makers, and so their near-term and medium-term binary equivalents provide a natural benchmark with

> 19The original approach splits trees based on a measure of node purity. This increases the probability that the same variables ends up at the majority of root nodes when _mtry_ increases, leading to high similarity between trees. Countering the correlation through randomization can produce stronger and more diverse trees that improve forest accuracy. This paper implements randomization of splits using the _extraTrees_ algorithm introduced by (Geurts et al., 2006). This selects cut-points at random, i.e., independently of the target variable. Key advantages are potentially increased forest accuracy due to lower tree correlation and faster runtime which makes tuning _mtry_ more practical and more interesting.

> 20In addition to its categorical ratings of actual food insecurity conditions, FEWS NET also reports a “near-term outlook” and a “medium-term outlook”. These outlooks represent preliminary assessments of the most likely food security ratings one and two reporting cycles ahead. The near-term outlook corresponds to either 3 or 4 months ahead while the medium-term outlook corresponds to either 6 or 8 months ahead, depending on the frequency of the reporting cycle which increased to 4 months in 2016. Recall that these outlooks may also follow livelihood zones, in which case we aggregated them to the district level. In an identical treatment to the outcomes, we also net the outlooks from the projected humanitarian assistance effects, and thus validate the binarized indicators purely on their ability to signal the need for action, disregard of whether that action was already anticipated. This is in line with the use case explored for model-based forecasts.

12