The key issue to be solved is what the ‘‘optimal’’ threshold level is. This is not trivial from a policymaker’s perspective. The difficulties arise because there is a trade-off between the “truepositive rate” (TPR) or “sensitivity”, i.e. the probability of signaling an insolvency event when the insolvency actually occurs, and the “false-positive rate” (FPR) or “1 - specificity”, i.e. the probability of signaling an insolvency event when there is no insolvency. The perfect signal would be one such that TPR equals 100 percent and FPR equals 0 percent. There is, however, a trade-off between the two which depends on the value chosen for the threshold. Setting the value of the threshold above which an insolvency event will be signaled at too low a level would trigger a high TPR (correctly classified insolvencies) but also a high FPR (too many false alarms). Conversely, setting too high a value would trigger a low TPR (large number of not called insolvencies) and a low FPR (low number of false alarms).

Setting the right value of the threshold depends on the shape of the policy maker’s utility function. If, for example, letting a bank go burst is very costly compared to the expected cost of implementing a policy measure, for example a recapitalization, the policymaker would be relatively averse to low TPR. If, on the other hand, the expected cost of implementing an intervention is instead very high relative to the expected benefit, then the policymaker would be relatively averse to high FPR.

Unfortunately, the utility function of the policy maker is not known. Moreover, it is likely that it varies over time and across countries. A way to account for the importance of the policy maker’s preferences without being forced to impose a specific functional form common to all countries is the use of the ROC curve.<sup>8</sup> The latter is the mapping between TPR and FPR for all possible thresholds. The ROC curve has, therefore, the nice property of providing a measure of the quality of the signal over the full range of possible utility functions. In particular, the area under the ROC curve (AUC) represents the relative forecasting success for true-positives reduced by the proportion of false-positives generated for any given forecasting threshold. An AUC of 50 percent indicates prediction no better than chance, while an AUC of 100 percent indicates perfect prediction.

To assess the predictive power of our EWS, we calculate both the AUC statistic for the fitted model (in-sample) and the AUC measure out-of-sample. For the latter, we conduct the validation test on the sub-period 2008-2012, which includes three insolvency episodes: two in Egypt and Kuwait in 2008, and one in Tunisia in 2010.<sup>9</sup> For this purpose, we estimated the baseline model over the sub-period ending in 2007 (training dataset) and calculated the AUC for the period 2008-2012.

Results show that our EWS estimated for the whole data set captures around 93.79 percent of the AUC. Similar results are obtained for the fitted model applied to the training dataset (AUC is 93.51 percent), whereas when applied to the validation data set (out-of-sample), the AUC is 92.06 percent. Overall, then, we conclude that our EWS for forecasting bank insolvency in MENA displays reasonably good predictive power both in-sample and out-of-sample.

> 8 See Hsieh and Turnbull (1996) for a general discussion of the ROC and Drehmann and Juselius (2013) for an application to banking crises.

> 9 Insolvency observations are, however, five as there are two cases of repeated insolvency.

9