First, if a “country record” flag is raised during the first half of project implementation, the odds of receiving an MS+ IEG outcome rating on average decline by about 15 percentage points (see annex table A.7) for PSM projects. This finding is particularly relevant since the country record flag does not reflect the performance of the PSM project in question<sup>73</sup> but the performance of the Bank’s overall lending portfolio in the respective country.<sup>74</sup> The country record flag was raised in about 17 percent of PSM projects<sup>75</sup> and thus should be considered as a telling, and relatively frequent, risk indicator. Second, TTL ratings of a PSM project’s progress toward reaching development objectives (PDO ratings) during the first half of project implementation predict project outcome ratings well. An increase of the _minimal rating_ used by TTLs for this measure by 1 point on the ordinal 6-point rating scale is on average associated with a 3.6 percentage point increase in the likelihood of receiving MS+ IEG outcome ratings (see annex table A.7). Finally, if a project receives an “at risk” status at least once during the first half of its implementation, its outcomes are on average about 6.2 percentage points less likely to be rated “moderately satisfactory” by the IEG (see annex table A.7).<sup>76</sup> One relevant conclusion from these findings for the Bank’s approach to monitoring risk in PSM projects is that the TTLs’ subjective assessment of PSM project performance and riskiness is a much more telling risk predictor than the more standardized risk flags used to monitor project implementation progress.<sup>77</sup> This is unsurprising in the sense that TTLs possess rich information about their projects and are best positioned to assess risk. But they may not have the opportunity or incentives to fully reveal these risks to management. This finding suggests that making sure that TTLs’ concerns are heard and that open conversations about risk are encouraged need to be core elements of the Bank’s approach to handling risk. These steps can be complemented but not substituted for by a system of systematically measuring specific risks (as is done by the risk flags). The “Operational Risk Assessment Framework” (ORAF) introduced by the Bank in 2011 points in this direction, offering task teams a way to systematize subjective risk assessments.

> 73 Three risk flags—“country record,” “country environment,” and “effectiveness delays”—do not capture project implementation performance. The other nine risk flags are linked to implementation performance. 74 More precisely, the “country record” flag gets triggered when the Operations Evaluation Department’s (OED’s) evaluations find at least one of three conditions to be true: (i) the net disconnect—a measure of realism of regional staff's portfolio performance assessments<sup>74</sup> is 20 percent or higher; (ii) disbursements of projects associated with an unsatisfactory rating (by OED) are 40 percent or more of completed projects; (iii) the Country Assistance Evaluation (CAE) has been less than satisfactory in the previous five fiscal years. The country record flag thus should be considered as a Bank-specific country-context measure that signals tough lending environments, rather than as a project-specific measure.

> 75 See Blum (2014) for detailed statistics. 76

> As a project’s “at risk” status is triggered when either its “implementation progress” (IP) or “development objective” (DO) rating is below moderately satisfactory, it is closely correlated with these two measures—and it is surprising that it has distinctive predictive power for a project’s final IEG outcome rating.

> 77 According to the Bank’s ISR guidelines, “the PDO rating is forward-looking, in that it assesses the likelihood that the PDO will be achieved.” More precisely, “for projects in the early stages of implementation, for which intermediate outcomes are not yet observable, the PDO rating is based on (i) implementation performance ratings and achievement of scheduled outputs and (ii) judgments about the likelihood that major risks—factors outside the control of the project—can jeopardize the achievement of the project’s outputs and/or outcomes.” In practice, the PDO rating is seen as the one risk indicator in the ISR that the TTL can use with great discretion to signal to management a subjective assessment of a project’s performance. Most of the other risk indicators—flags and the IP rating—are triggered based on more mechanical/standardized criteria with less discretion.

37