# Annex B: How is the IEG database built? Why are these ratings useful?

The evaluations reported in this database are constructed through two methods: ES/EVM and PPAR. The ES/EVM, evaluation summary or memoranda, are project evaluations prepared and recorded by the responsible Bank department (not by IEG) at the end of the project (reported in the ICR – Implementation Completion Report). These data are then reviewed by the ICR Review board (this procedure was established in 1995), which is an intermediate World Bank-IEG committee which seeks to independently verify the findings of the ICR. The problems behind EVM evaluations are linked with the self-assessment produced by projects responsible, which may create an upward bias in the data. On the contrary, the Project Performance Assessment Report (PPAR) offers a much stronger picture of the evaluation. These are developed by IEG, which independently and randomly samples 25% of World Bank projects per year, sends its experts on field to evaluate the results of these projects and create independent evaluations Though these ratings may not be perfect, we feel that they contain some important pieces of information, which may provide useful policy recommendations. One could ask whether these evaluation ratings are realistic? We feel they are pretty much. The department within the World Bank in charge of their collection, IEG, spends substantial resources in reviews and quality controls, employs very experienced evaluators, has very strict confidentiality and anti-conflict of interest norms. In addition to this, the institutional architecture of the World Bank structures IEG as a special and independent department, which reports only to the Board of Directors (the decision-making entity of the World Bank) and not to single departments/vice presidencies, it does not share personnel, resources or initiatives with other parts of the Bank; surprisingly enough IEG does not even share the location with other departments and is currently placed in a different building. It may also be asked whether these evaluations are taken seriously? Again, after analyzing internal reports and discussions, we feel they are. IEG creates a whole range of products that comment on the evolution of World Bank’s projects and these reports are mostly based on the ES/EVM and PPAR data and the project reports behind their construction. Inside the World Bank there is a remarkable attention to the evolution of the ratings, so far that in 1992 after a long-lasting decline in project performance ratings, the Wapenhans Report (Wapenhans 1992) called for immediate changes in the operational structure of the Bank, as it “identified poor project monitoring and supervision, as well as a lack of focus on project outputs and development impact, as contributing to poor project portfolio performance” (Sigurdsson and Schweitzer 1995). Another question which the nature of this data may spark is whether these evaluations are reliable. This may present two types of concern. The first and more general one regards how we define successful a project. It is clearly a micro-definition, which heavily depends on the project; in fact its outcome variable measures the “extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved, efficiently”<sup>19</sup> . Therefore, a project is successful if it reaches the stated objectives. This may lead to think that project results may depend on how carefully their objectives are stated, for example a project may perform exceptionally well given extremely low goals or viceversa. The World Bank addresses this issue substantially in order to avoid such problems, by requiring an intense project review process within sectors, questioning whether the project is sufficiently ambitious, feasible and meets the poverty-fighting objectives guiding the World Bank mission. In order to attain comparability standards several departments are involved in a collective review of each project: the legal Vice Presidency, the procurement Vice Presidency, the regional Vice Presidency, the sector manager within the regional Vice Presidency, and for complex loans a whole committee, including the Network anchors<sup>20</sup> and other Vice Presidencies. A second concern may regard the credibility of these ratings, especially for what concerns the ES/EVM relative to PPAR. In their paper Denizer, Kaufman and Kraay (2011) do not find significant differences between the two evaluation types and their dummy capturing this difference is insignificant. In our study, we adopt different variables from their paper, still we do not find significant differences.

> 19 Refer to

> 20 By Network Anchor it is intended the “sectors” in which the Bank’s work is developed, these are: Agriculture and Rural Development, Energy and Mining, Environment, Global Information/Communications Technology, Transport, Urban Development and Water, Economic Policy, Education, Financial and Private Sector Development, Financial Sector, Health Nutrition and Population, Poverty Reduction, Public Sector Governance, Social Development and Social Protection.

<u>http://web.worldbank.org/external/default/main?theSitePK=1324361&pagePK=64253958&contentMDK=20789730&menuPK= 5039271&piPK=64252979 http://web.worldbank.org/external/default/main?theSitePK=1324361&pagePK=64253958&contentMDK=20789685&menuPK= 5039271&piPK=64252979</u> 32