which to compare the model-based forecasts. Because FEWS NET data do not assign a probability to future outcomes, we only focus on the _LA_ loss function of Equation 1. Table 3 contains the results.

The most striking feature of Table 3 is the extreme imbalance between false alarms – an FPR of around 2% – and failure to predict – an FNR of over 74%. This clearly indicates that the historical performance baseline is characterized by a strong aversion to predicting transitions into food crises that subsequently do not occur. In the context of our evaluation framework, this can be interpreted as evidence of a very high implicit weight on false positives and a low weight on false negatives, i.e. a low value of w.<sup>21</sup> The low false alarm rate does not mean that those warnings that were raised, did also most likely predict a transition correctly. To the contrary, only 35% of the projected transitions into crisis were followed by actual transitions. Comparing the frequency of transitions into crisis in both the historical assessments and outlooks also highlights the conservative nature. Although transitions into crisis account for 5 _._ 3% of the historical assessment sample, only 3 _._ 2% of the binarized outlooks projected a transition, see Table B.1 in supplementary Appendix B.

Table 3: Historical performance baseline calculated from binarized FEWS NET outlooks.

||**Error **|**Types**||**_LA_**||
|---|---|---|---|---|---|
||**FNR**|**FPR**|**w=1/3**|**w=1/2**|**w=2/3**|
|**Near-term**|74.3%|1.4%|25.7%|37.8%|50.0%|
|**Medium-term**|79.4%|2.2%|27.9%|40.8%|53.7%|

**FNR** and **FPR** of binarized historical FEWS NET outlooks in the near-term (3-4 months, validated against outcomes one reporting cycle later) and medium-term (6-8 months, validated against outcomes two reporting cycles later) and three weighted averages ( **_LA_** ) with weights **w** and (1- **w** ) for prediction errors in the positive (negative) class. Both the binary outcomes and outlooks are netted of humanitarian assistance effects. The validation was performed at the admin2 level, and only for observations that were preceded by non-crisis levels of food insecurity. This specifically describes prediction performance for new outbreaks of food crisis before they occur. The validation sample spans all available projection-outcome pairs from July 2009 - February 2019.

Table 4: Cross-validated performance of predictions from the wide Random Forest.

||||**Error **|**Type**||||**Bal**|**anced M**|**etrics**|||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|||**FNR**|||**FPR**|||**_LA_**|||**_LB_**||
|**w**|**1/3**|**1/2**|**2/3**|**1/3**|**1/2**|**2/3**|**1/3**|**1/2**|**2/3**|**1/3**|**1/2**|**2/3**|
|**h=4**|19.7%|16.5%|16.0%|6.8%|8.2%|8.4%|11.1%|12.3%|13.5%|0.25|0.29|0.31|
|**h=8**|19.1%|16.6%|16.6%|7.3%|8.5%|8.5%|11.2%|12.5%|13.9%|0.27|0.30|0.33|
|**h=12**|20.9%|20.9%|9.9%|10.0%|10.0%|19.0%|13.6%|15.5%|13.0%|0.30|0.34|0.35|

The first six columns report false negative rates and false positive rates for models optimized for the indicated value of **w** . The next six columns report weighted averages and weighted Log Loss values ( **_LB_** ) with weights **w** and (1- **w** ) for prediction errors in the positive (negative) class. The statistics have been cross-validated for **h** =4-, 8- and 12-month ahead forecasts of crisis outbreaks. The validation was performed at the admin2 level, and only for observations that were preceded by non-crisis levels of food insecurity. This specifically describes prediction performance for new outbreaks of food crisis before they occur. The validation sample spans from July 2009 - February 2019.

> 21Recall that unbiased Log Loss sets w to the frequency of the positive class in the validation sample, 5 _._ 3% on average in random draws, hence an unbiased model that assumes both the false positive rate and false negative rate carry the same cost, would assume a value of around w _∼_ 0 _._ 05. The binarized outlooks produce even fewer alarms, at a rate of 3 _._ 2%, meaning that implicitly w _<_ 0 _._ 05. At, w= 0 _._ 03 the rate of false alarms is weighted 30 times as much as the rate with which anticipating crises fails in a weighted cost function like _LA_ .

13