- Only in the last quarter of the project’s life (measured by time or by disbursement amount) does the DO rating reach the prediction power of the model using only the CPIA and task team leader quality variables. Until then, it is lower than or equal to 20 percent.

- For EAP, the predictive performance of the ISR DO rating was even lower.

- Combining the first model with the ISR DO rating during each quarter of the life of the project leads to a predictive power of slightly above 60 percent in the last quarter.

- Using the IP rating instead of the DO rating improves the prediction model, and using the two together further enhanced the results.

- Applying the model to the EAP portfolio as of July 1, 2014, yielded a predicted rate of unsatisfactory outcome of 17 percent.

The two studies clarify the relative importance of country and project variables, including flags and other indicators for predicting project outcome, but they have several limitations. First, since the model significantly underestimates current IEG rates of unsatisfactory outcomes for EAP, now close to 30 percent, it will need to be re-run and calibrated using more recent data. Second, the quality of the task team leader is a difficult, sensitive, and controversial variable to estimate fairly and regularly. Third, the model still only predicts about 40 percent of the unsatisfactory outcomes.

# III. Foundation for a New Outcome Prediction Tool for Moderately Satisfactory (or Higher) Rated Projects

Given the limitations of existing monitoring tools and predictive models, it would be advantageous to find a simpler approach that could yield similar or better results, by using real-time data, such as the IP and DO ratings, even if it meant moving away from a pure prediction model, i.e., a model that would mostly include variables already known at the project start. Hence, EAPDE conducted a thorough analysis of the projects rated less than moderately satisfactory and particularly disconnects in search of a tool that would predict whether a project rated moderately satisfactory would be downgraded on completion.

## Methodology for analysis

Since the review started out of concern about the sharp decline in IEG rates of satisfactory outcomes in the latest set of projects reviewed, it was logical to use the set of projects that exited in FY12 and FY13 for analysis. Since the focus was on identifying factors and patterns contributing to the increase in unsatisfactory outcomes, the review concentrated on projects with unsatisfactory ratings, particularly those with disconnects.

The convenience sample included all 62 investment projects that exited in FY12 and FY13 with ICR reviews posted on the IEG site as of November 15, 2014. In accordance with IEG review policy, the sample included all projects financed by IBRD or IDA as well as Global Environment Facility, State and Peace-Building Fund, and recipient-executed trust funding with commitment amounts larger than $5 million.

The IEG ICR reviews were first examined to identify patterns in the country and network or sector breakdown. Outcome results broken down by country were compared with corresponding CPIA figures and with candor gaps and recent series of ratings (Annex 1, Table 1). Since IEG uses four dimensions to 6