In this discussion we have emphasized the predictive power of the ISR-DO rating because it conceptually corresponds most closely the assessment of the development objective of the project that IEG performs in its evaluations. However, TTLs are also asked to flag difficulties in project implementation using the ISR-IP flag in each ISR. It is possible that this flag brings additional information about eventual project outcome ratings by IEG. To investigate this possibility, we reproduced the analysis in Figure 2 and 3, for two different variants. In the first, we simply replace the ISR-DO rating with the ISR-IP rating. In this case, we find that the predictive power of the ISR-IP rating is slightly better than that of the ISR-DO rating. For example, in the out-of-sample predictions in Figure 2b, the ISR-IP flag in the last quarter of project life as measured by disbursements (time) correctly predicts 48 percent (44 percent) of unsuccessful project, while for the ISR-DO flag the corresponding figure is 37 percent (42 percent). In the second variant, we constructed a hybrid of the two flags which took the value 1 if either the ISR-DO flag or the ISR-IP flag were raised. In the out-of-sample predictions in Figure 2b, the hybrid flag in the last quarter of project life as measured by disbursements (time) correctly predicts 42 percent (53 percent) of unsuccessful projects. However, our conclusion that the prediction model does a better job of predicting project outcomes relative to the ISR flags in the first half of project life remains true for either of these two variants.

# **6.  Integration with Regular Portfolio Monitoring**

In this section we illustrate how these predictions can be integrated into regular monitoring of the World Bank’s portfolio of active projects, in order to better identify projects at risk of not having satisfactory outcomes. We first re-estimate the basic prediction model using data from all evaluated projects 1995-2012. The results are shown in Table 4. The results are broadly similar to those in Table 2, except that project length falls just short of significance at the 5 percent level, while preparation time is now significant. We then apply the coefficient estimates from this model to generate predicted probabilities of unsatisfactory project outcomes in the active portfolio.<sup>7</sup> We calibrate these predictions

> 7 The active portfolio is defined by the set of operations--financed by IBRD, IDA, full sized GEF (>$1 m), large Recipient Executed Trust Funds (> $5 m), Special Financing, and Montreal Protocol TF--that were open on June 30, 2013.  Some projects in the active portfolio have missing data for some of the variables included in the prediction model.  In order to be able to generate predicted outcomes for all projects, we impute these using the sample means of the same variable for all other projects.  This affects relative few projects, except in the case of (i) the TTL record variable, which is not available for about 10 percent of projects (reflecting the entry of “new” TTLs into the portfolio, who do not have any IEG evaluations of their past projects in the historical data on which the TTL record variable is based), and (ii) time from approval to effectiveness, which is not available for the roughly 20 percent of projects in the active portfolio that have been approved but are not yet effective

14