as a potentially powerful way to ensure more flexibility and experimentation in the implementation process.

While data are unavailable on many important reform process characteristics—such as the breadth or intensity of stakeholder engagement—it is useful to ask which of the currently collected risk indicators actually predict IEG outcome ratings.

## **3.8 Project Management Factors**

Project management factors include basic project characteristics, such as the committed lending amount, preparation and supervision costs, project preparation times, and so on.<sup>18</sup> Such factors are distinct from process or risk indicators, in that they are not explicitly designed to indicate risk, and they may be observed before project implementation starts.

Several of these factors are potentially useful predictors of project performance. The time required for project preparation<sup>19</sup> could indicate particularly careful preparation—but could also indicate a particularly controversial, challenging project. A long delay before the project actually becomes “effective” may indicate weak client commitment to the project.<sup>20</sup> Regarding loan size, smaller projects may get less highlevel attention than large projects, both on the client and on the Bank side.

# **4 THE DATA**

This paper analyzes project-level data from the World Bank’s project management information system as well as country-level data from various sources. This section describes the project sample underlying this paper, the data employed, and associated measurement issues.

## **4.1 The Project Sample**

− The sample used for this paper includes all World Bank supported investment lending projects<sup>21</sup> approved between FY 1990 and FY 2013 that had been closed and evaluated by the IEG at the time of the

> 18 Identifiers of the Bank teams or task team leaders (TTLs) responsible for project preparation (and their performance track record) are not included in this paper. While they are observable in principle, data on preparing TTL identities have only been captured systematically since 2005. Such data are therefore not available for most of the project universe underlying this paper. It is important to note that Denizer, Kaufmann, and Kraay (2011) find that TTL identity and performance are significant predictors of project performance.

> 19

> 20

> The time required for project preparation is a reductive, but possibly telling, sign of the nature of this process. A first measure of preparation time is the “time to approval,” that is, the time it took from the project concept note review (PCN) meeting to project approval by the Bank’s Board of Directors. Above average times to approval could indicate (i) a particularly carefully prepared project or (ii) a particularly complex project that may be challenging to implement, or (iii) a project that is particularly controversial and therefore took a long time to agree on.

> One key step required between project approval and effectiveness is that the client government sign the loan agreement. A delay in this step may reflect a lack of urgency or unresolved points of contention. “Time to effectiveness” may thus be a signal of low client government commitment to the envisaged project, an important but hard to observe driver of project performance.

> 21 Projects were included in the sample only if (i) they were approved between FY 1990 and 2013, (ii) their “lending instrument type” was classified as “investment” lending, and (iii) their committed amount was greater than zero.

16