civil liberties in anticipation of a challenging reform environment, the estimates in this review would underestimate the detrimental effect of limited liberties on the success of PSM reform.<sup>34</sup> Despite these caveats (and as argued in section 3.2), IEG outcome ratings have the major advantage of not depending on normative claims about what PSM reforms should achieve across countries. Rather, they provide a measure of efficacy<sup>35</sup> in implementing a PSM reform plan that was agreed upon by and deemed acceptable by both the client country government and the Bank’s Board of Directors.

# **4.4 Country Context Measures**

The country-level data employed in this review come from multiple sources that include the Freedom House measures of Political Rights and Civil Liberties; the Polity IV institutionalized autocracy, anocracy, democracy, and combined polity scores; the Share of Programmatic Political Parties from the Database of Political Institutions; the CPIA 12-16 Ratings; the International Country Risk Guide (ICRG) Bureaucracy Quality Rating; as well as several variables from the World Development Indicators (WDI) data sets.<sup>36</sup>

# **4.5 Process and Risk Measures**

Process or risk indicators routinely collected in implementation supervision reports (ISRs) include a set of 13 (binary, on/off) risk “flags,” for areas such as financial management, disbursement delays, environmental safeguards, and so on. To limit the number of covariates included in estimations, only the flags relating to the country environment, country record, project management, safeguards and counterpart funds—as well as the total number of risk flags raised—were included in the final estimation.<sup>37</sup> In addition, ISRs include two markers that TTLs may use to indicate how well a project is going: (i) “implementation progress,” that is, whether project activities are being implemented as planned; and (ii) “development objectives,” or how well the project is expected to achieve its ultimate development objectives. It should be noted that of these, the first is not entirely at the discretion of the TTL—if three or more risk flags are raised, it is automatically set to “moderately unsatisfactory” or less.<sup>38</sup>

> 34 A third potential limitation is that the IEG ratings are ultimately subjective assessments conducted by IEG staff. But it can reasonably be assumed that these ratings do not systematically suffer from bias, given the IEG’s institutional setup as an independent agency within the Bank. If this is true, subjectivity is merely a source of noise and not a concern for the purposes of this review.

> 35 Isham, Kaufmann, and Pritchett (1995) introduce this term.

> 36 Many of these measures have limited availability or may have questionable validity or consistency over time. Please refer to Blum (2014) for a more detailed discussion of these issues.

> 37 This selection is based on theoretical expectations of the relevance of the respective risk flags, the frequency of their usage, and their suitability for improving the fit of the estimation model.

> 38 Based on these risk ratings, projects are marked as “at risk” or as “proactivity” projects that require attention/action. Actions taken in response to these risks are also recorded, and required to move a project out of “at risk” status. “At risk” and “proactivity” statuses are also included as covariates.

21