ISR ratings for IP and DO also underestimate projects likely to be rated moderately unsatisfactory During routine project supervision projects are rated on their IP and DO. A moderately unsatisfactory rating is given to projects that fall short on performance standards and are therefore likely to fail at meeting their development outcome at exit. These projects are then referred to as “Problem Projects.” Table 1 shows the percentages of projects rated moderately unsatisfactory for IP, DO, or both as of the end of 2015. These percentages are believed to be substantially below the failure rates of these projects at exit. As previous studies have shown, this is a very consistent and stable occurrence across both time and institutional levels, since it is rooted in the over-optimism or lack of candor of task team leaders when they self-report on their project in the ISR report.

**Table 1. Percentage of Moderately Unsatisfactory or Lower Ratings for DO and IP, all Investment Projects as of December 31, 2015**

||**DO**|**IP**|**DO or IP**|
|---|---|---|---|
|**Bank**|15%|20%|21%|
|**EAP**|15%|21%|22%|

# II. Existing Approaches to Predicting Performance

As the preceding discussion shows, the current tracking mechanisms are not very effective at predicting whether a project rated moderately satisfactory for most of its life will be downgraded on completion. But those are not the only ways that the Bank tries to manage its risks.

## Risk at entry – difficult to ensure update during implementation

One method for managing risk identifies and defines the level of risk at entry, that is, during the preparation and approval phase of the project, using relatively well defined criteria and methodologies. This approach has been useful at the preparation stage to identify comprehensively the risk profile of the project, and thus the type of processing and level of resources required. It has proved much less useful after Board approval, probably because it is difficult to get the task team leader to regularly and precisely update this risk rating during project supervision.

## Risk under implementation – a system that has resulted in a perverse incentive to manage flags

Another approach has identified a risk profile of the project under implementation based on flags, or warning indicators. These have included self-reported indicators, such as for DO and IP, as well as system-based indicators, such as disbursement delay calculated by comparing the actual and the initial or revised disbursement schedules. The full set of indicators, currently numbering about ten, are being input in the ISR report and are tracked in the project monitoring system; several of them are also monitored at the corporate level (for example in the Corporate Scorecard, memoranda of understanding, and senior management monthly monitoring meetings).

Until about 2007 the flag system (then consisting of 12 flags) was used to identify and track “potential problem projects” that were not rated moderately unsatisfactory or lower for IP or DO during implementation. This led some teams to avoid the appearance of failure by finding ways to avoid rating 3 flags as less than satisfactory. This undercut the ability of the system to detect problem projects and the system was dropped, though 10 of the flags remain in use.

4