14

# _Assessment of prediction performance_

The predictability assessment of the model is done by using the regression results in Column 4 of Table 1. This paper uses Brier’s Quadratic Probability Score (QPS), which is defined as: where _pn_ is the predicted probability of credit-less recovery in episode _n_ , and _yn_ is the actual recovery in episode _n_ (i.e., the dependent variable in Table 1, a dummy variable taking 1 if a recovery is credit-less). _N_ is the total number of episodes ( _N_ = 150). QPS ranges from 0 to 1 and a score of 0 implies perfect accuracy. For the estimation model in Table 1, the computed QPS is 0.18; that is, the prediction performance of the model is quite accurate.

In addition, this paper looks at individual countries in ECA to see whether the model correctly specifies the recovery pattern these countries have experienced. It focuses on episodes related to the 2008–09 global financial crisis and covers 17 ECA countries for which the data required for probability calculations is available. In order to evaluate the performance for individual countries, this paper sets two thresholds: 0.5 and 0.3. The former is taken from Canova (1994) and also used in the analysis done in Bijsterbosch and Dahlhaus (2011), and the latter is the average share of credit-less recoveries in our sample for all emerging market economies. Using these two thresholds, this paper investigates the presence of the type-I and type-II errors (i.e., false positive and false negative, respectively). In this context, the type-I error means that a recovery is predicted as credit-less though in reality it is not, while the type-II error means that a recovery is not predicted as credit-less but turns such a type of event. From the perspective of the adequacy of policy responses, type-II errors can be viewed as more costly; that is, not predicting a credit-less event might lead to unwanted lags in the policy response.

Figure 5 shows the predicted probability for each of recovery episodes in ECA countries over the period of 2009–10. High probability of credit-less recoveries is predicted for the Baltic states, especially Latvia, as well as Russia and Ukraine. The results are largely driven by the fact that these countries experienced large contractions in economic activity and/or a sharp depreciations of their currencies. Recoveries are predicted as credit-less if the probability is above a threshold of 0.5 (i.e., black bars) and as credit-with if it is below 0.3 (i.e., light gray bars). Any episodes located in between 0.3 and 0.5 are treated as borderline cases. Out of 17 episodes (or, countries), 9 episodes are predicted as credit-less recoveries and 5 are not predicted as credit-less recoveries. By combining this figure with the actual recovery patterns experienced by these countries (that is, the evidence on credit performance), it is found that there are three type-I errors (i.e., Armenia, Georgia and Turkey) and one type-II error (i.e., Kyrgyz Republic).<sup>14</sup> The existence of only one costly mis-prediction (i.e., a type-II error) is somewhat comforting from a modeling perspective.

> 14 A caveat is that the calculations are made in-sample.  Still, given that ECA countries represent a small part of the recovery sample, it is safe to say that the risk of an in-sample bias is low.