The statistical model-based predictions we develop have the potential to improve on this baseline in several ways. We explore this using Table 4, which provides cross-validation results for 4-, 8- and 12-month ahead forecasts. Note that the 4- and 8-month forecast horizons are roughly comparable to the near- and medium-term outlooks considered by FEWS NET.

First, when we set a conservative value of w=<sup><u>1</u></sup> 3<sup>toimposeaheavierpenaltyonfalse</sup> alarms, the FPR increases slightly relative to the performance baseline produced with binarized FEWS NET outlooks, to around 7% for the 4- and 8-month forecast horizon. However, this comes with a very large reduction in the FNR to below 20% which is substantially down from the baseline rates of 74% and 79%. Taken together, the slightly higher FPR and much lower FNR results in an improvement in _LA_ from the baseline value of around 26% using binarized FEWS NET outlooks, to around 11% for the model predictions.

Second, the model-based approach allow us to fine-tune predictions for different use cases corresponding to different weights in the prediction loss function. As we have discussed, the severe costs of food crises suggest that for some policy purposes, a greater penalty should be imposed on failure to predict crises, i.e. that a higher value of w would be warranted. The FEWS NET outlooks offer just one fixed set of forward-looking ratings, and increasing the weight on FNR in the prediction loss function in Table 3 sharply increases _LA_ , indicating a sharp decline in predictive performance with regard to the less conservative forecasting objectives. For example, for the medium-term outlook, _LA_ increases from 28% to 54% when w increases from<sup><u>1</u></sup> 3<sup>to</sup><sup><u>2</u></sup> 3<sup>.Incontrast,themodelpredictionsmaintaincomparablepredictive</sup> performance across the range of weights, as evidenced by minimal changes in the weighted performance metrics.

Third, models can learn different predictive associations depending on the forecast horizon and tailor predictions to the lead-time that is required. The complexity associated with unlocking funds and investing in prevention implies that near-term forecasts have lower utility than medium-term forecasts when equally accurate and so it would be helpful to predict further into the future with similar accuracy. The FEWS NET outlooks offer projections that run roughly 8 months into the future which, when binarized, score an _LA_ of 28% for the conservative weight of w=<sup><u>1</u></sup> 3<sup>.We assessed the ability to predict 12 months ahead,</sup> and find that the model maintains an _LA_ below 14%, still roughly half of the medium-term performance baseline.

Fourth, models have the ability to predict the probability of an event which means they can rank districts according to risk. _LB_ measures how well the predicted probabilities align with the outcomes and rewards predictions that trend in the right direction. The minimum value of _LB_ depends on the degree of noise in the target variable but it is well-known that an uninformative model, that predicts only the correct average probability, attains a Log Loss of _._ 69.<sup>22</sup> Hence, any value below _._ 69 for any value of w implies that the model not only

> 22As a reference, it can be shown that in a system, where you would know all the possible information about its future state other than some unknown internal level of randomness, so that, given that all model parameters are at their correct values, an outcome can still be different than predicted. Say for example that in this system the best one can do is predict 0.1 when the outcome is 0, and .9 when the outcome is 1, and that for one out of ten predictions the outcome falls into the other category. Then, Log Loss attains a minimum of approximately _._ 33.

14