metadata: authors: 0: author: Di Maro,Vincenzo
1: author: Legovini,Arianna
2: author: Piza,Caio
count: World
docna: 0: docna: Impact evaluation helps deliver development projects
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Educational Sciences,HIV AIDS,Cholera,Malaria,Communicable Diseases,Leprosy,Transport Services
teratopic: Law and Development
abstracts: cdata!: Does research add value to aid?
            Specifically, does impact evaluation research help or hinder
            the delivery of development projects? This paper analyzes
            the question by constructing a new data set of 100 impact
            evaluations and 1,135 projects approved by the World Bank
            between 2005 and 2011. The analysis finds that the delivery
            of projects with impact evaluation is significantly
            timelier: common delays are avoided and the gap between
            planned and actual disbursements is reduced by half.
            Evidence-based mid-course corrections, a clearer
            implementation road map, strengthened capacity on the
            ground, and observer effects are possible channels to
            explain the results. Hopefully, this analysis will stimulate
            discussion over the optimal balance between project
            financing and the impact evaluation research needed to
            deliver development outcomes.
repnme: repnme: Impact evaluation helps deliver development projects
display_title: Impact evaluation helps deliver
            development projects
listing_relative_url: /research/2015/01/23173058/impact-evaluation-helps-deliver-development-projects
topicv3: Transport m1326661 135,Malaria m1327042 1420,Education m1327048 470,HIV/AIDS m1327522 1416,Health m1504661 3480
publishtoextweb_dt: 2015-01-01T00:00:00Z
historic_topic: Law and Development
no_of_pages: 40
author_keys: 0: author_key: 805510
1: author_key: 85205
2: author_key: 1239892
model_extractions: WPS7157 Policy Research Working Paper 7157 Impact Evaluation Helps Deliver Development Projects _Arianna Legovini Vincenzo Di Maro Caio Piza_ Development Research Group Impact Evaluation Team January 2015
Policy Research Working Paper 7157

# **Abstract**

Does research add value to aid? Specifically, does impact evaluation research help or hinder the delivery of development projects? This paper analyzes the question by constructing a new data set of 100 impact evaluations and 1,135 projects approved by the World Bank between 2005 and 2011. The analysis finds that the delivery of projects with impact evaluation is significantly timelier: common delays are avoided and the gap between planned and actual disbursements is reduced by half. Evidence-based mid-course corrections, a clearer implementation road map, strengthened capacity on the ground, and observer effects are possible channels to explain the results. Hopefully, this analysis will stimulate discussion over the optimal balance between project financing and the impact evaluation research needed to deliver development outcomes.

This paper is a product of the Impact Evaluation Team, Development Research Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at alegovini@worldbank.org, vdimaro@worldbank.org and ctpiza@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# IMPACT EVALUATION HELPS DELIVER DEVELOPMENT PROJECTS

Arianna Legovini, Vincenzo Di Maro, and Caio Piza<sup>*</sup> JEL No. O01, O12, O19, O02, D04, H43, C93, F61 Keywords: Impact Evaluation, Development Effectiveness, Development Policy, Experiments, Projects

* Arianna Legovini is the Manager of Development Impact Evaluation in the Research Group of the World Bank, and Vincenzo Di Maro is a senior economist and Caio Piza is an economist in the same group. We are especially grateful to Aart Kraay, Guadalupe Bedoya, Florence Kondylis, Daniel Rogger, and Daniel Stein for their detailed comments on an earlier draft. We thank Pavel Luengas and Susumu Yoshida for their assistance in collating project-level data from the World Bank’s databases and providing research assistance.
# 1. INTRODUCTION

Development effectiveness has been at the center of the economic debate since at least the establishment of the Bretton-Woods institutions. With $16 trillion in aid delivered since then, this is by all means an important issue.<sup>1</sup> Despite all efforts, however, the knowledge gap on what really works in development remains large. The literature investigating the relationship between country growth and aid flows has for the most part failed to provide satisfactory answers to the question of whether aid is effective and under what conditions. This is in part because aid measured in dollars and cents implicitly assumes that aid is a transfer of resources, a noisy proxy for the bundle of technical and financial assistance through which aid is delivered.

Far from being a direct transfer, aid is delivered through development projects and general budget support, each representing historically 86 and 14 percent of the total.<sup>2</sup> Both of these instruments bundle technical assistance with financial flows, the former affecting the effectiveness of the latter. Recent evidence (Denizer et al. 2013) finds that most of the variation in aid performance is explained by project characteristics (and not by country conditions). This suggests that the relevant question may not be whether aid is effective but what quality of projects and project characteristics make aid effective.

With $3.5 trillion committed to developing countries since 1970,<sup>3</sup> the World Bank is a large development actor, and one that invests much effort in the quality of projects, from peer reviews to quality enhancing processes. Still, about a quarter of projects are assessed to have been of unsatisfactory quality at entry<sup>4</sup> and a quarter rated unsatisfactory at exit. Projects also experience implementation delays with lags of six to seven quarters between project approval and first disbursement and continued gaps between actual and planned disbursements thereafter (World Bank 2014a, 2014b).

In 2005, the Bank started an institutional effort to do impact evaluation (IE) to generate knowledge in the context of national policies and programs in developing countries (Figure 1). This is also when the Bank started tracking the link between IE and its projects.<sup>5</sup> Over the next few years, and starting

> 1 Net official development assistance and official aid received (current U.S. dollars). Source: data.worldbank.org.

2 http://aiddata.org.

3 IBRD loans and IDA credits (DOD, current U.S. dollars). http://data.worldbank.org.

> 4 Quality at entry is assessed by the Quality Assurance Group (QAG) on the basis of a sample of projects approved by the Board in the previous fiscal year. Reports are shared with management for remedial action.

> 5 Before 2005, there were one to five studies a year evaluating World Bank projects. After 2005, the Bank started a concerted effort to evaluate Bank projects and track the link between projects and IEs. The number of IEs related to projects jumped to about 20 a year after 2005.

2
from a very low baseline, IE came to cover about 10 percent of the Bank’s projects and establish, through counterfactual analysis, the causal pathways of what policies financed by these projects work and why.<sup>6</sup> The Bank’s impact evaluations involve research and field staff working with the project and government teams to introduce structured and iterative learning in project operations. They aim to inform policy design, identify policy-relevant research questions, run experiments, and collect and analyze data, to feed evidence back into policy decisions during implementation. At their best, impact evaluations are a framework of collaboration for the generation and real-time use of policy-relevant evidence.

This study investigates the role of these impact evaluations on the quality of projects. For this purpose, we assume that we can summarize the quality of projects by three conditions: (i) the problem/goal is correctly identified, (ii) an appropriate course of action and financing is agreed to, and (iii) project activities are implemented as planned. While impact evaluation can potentially influence (i-iii), it is hard either to measure or find good proxies for (i) and (ii). We thus restrict our analysis to (iii), which is measurable and a necessary condition for projects to achieve their impact.

While measuring the link between IE and development outcomes of projects is of great interest to the development community, it is at best challenging. There is plenty of evidence established on a case-by-case basis,<sup>7</sup> but estimating the impact of IEs in general requires a valid counterfactual and a common impact metric across the policies being evaluated. Constructing such a counterfactual seems unfeasible.

Our second-best, but tractable, approach is to investigate the link between IE and project implementation. To do this, we construct a panel using administrative data on the whole portfolio of World Bank impact evaluations and projects approved during 2005–11. Linking information on projects’ disbursements with information on impact evaluation allows us to explore the relationship between IE and project implementation using 100 impact evaluations and 1,135 projects implemented in 114 countries in 11 sectors during 28 quarters.

We use project disbursements as a measure of implementation. This is because disbursements are made against evidence that the agreed activities are being implemented by the relevant government and supervised by the Bank project staff. We use three specific quarterly measures: (i) cumulative disbursements relative to loan size, (ii) the gap between planned and actual disbursements relative to loan size, and (iii) the ratio of project to average disbursements for all projects in the relevant sector. To our knowledge, this is the first quantitative study aiming at identifying the effect of IE on the implementation of development projects.

> 6 In addition to IE, all Bank projects must complete an implementation completion report. About a third of them are also subject to project evaluation by IEG. These assessments are descriptive in nature, not causal.

> 7 Examples from World Bank IE work are Bruhn et al. (2013), Brune et al. (2011), Friedman et al. (2010), Kondylis, Jones, and Stein (2014).

3
To control for project selection into IE, we include both institutional and idiosyncratic factors, assuming the confounders are observed or time-invariant. These factors range from time, sector and country effects, project and project leader fixed effects, and project leader changes. If project leaders of potentially high-disbursing projects are more likely to select into IE, then projects with IE may disburse faster partially due to the project leader decision to adopt an IE. If, on the other hand, project leaders select into IE because their projects need additional help and support, then projects with IE will disburse at a slower pace. Our data provide some support for the latter case.

We find that impact evaluations have a large and significant effect on the rate of disbursement of operations, reducing the gap between planned and actual disbursements and increasing disbursements relative to average disbursements in the typical project in the same sector. By averting common delays in the early stages of the project cycle, impact evaluations might make less relevant the need for teams to push disbursements out in the later stage of project implementation.

We conclude that projects with IE are more likely to implement their activities as planned and, in so doing, may be more likely to achieve their objectives.<sup>8</sup> Furthermore, our results support the idea that project financing and IE research are complementary outputs of the World Bank. This might have implications for the use of IE as an operational tool to improve delivery of projects, and for weighing the costs of IE research against the benefits on the Bank’s project portfolio. For development organizations like the World Bank, an optimal balance between project financing and IE research may help them reach their efficient frontier. More generally, we hope our results may contribute to shifting the discussion away from just questioning the effectiveness of aid to a discussion on constructive ways to make aid more effective.

Finally, we consider the potential channels through which World Bank impact evaluations can affect project implementation. We speculate that these channels include:<sup>9</sup> (a) better planning and evidencebase in project design, (b) greater implementation capacity due to training and support by research team and field staff, (c) better data for policy decisions, and (d) observer effects and motivation. These channels are not necessarily specific to IE research and learning more about them could inform development policy more widely. Finally, we discuss the limitations of our analysis and outline an initial agenda for future research.

The remainder of our paper is organized as follows. Section 2 summarizes the literature. Section 3 describes the institutional background and the data. Section 4 explains the analytical strategy. Section 5 summarizes the results. Section 6 concludes.

# 2. LITERATURE REVIEW

> 8 Here, we are assuming that assumptions (i) and (ii) above are met; that is, (i) the problem/goal is correctly identified and (ii) an appropriate course of action and financing agreed to. 9 See also the discussion in Legovini (2010).

4
Most of the evaluation literature has focused on assessing the impact of interventions. This paper aims to evaluate the impact of using evaluation itself on the implementation of projects. While a vibrant literature exists on the question of whether aid is effective (see Temple 2010 for a review), we could find very little on the question of how to make aid effective. On the question of whether IE research helps deliver development aid, the contributions in the literature are all theoretical or descriptive. The Independent Evaluation Group report on IE bases its findings on a beneficiary assessment of completed IE studies; the results are self-reported and no comparison group is provided (World Bank 2012b). The 2011 _DIME Progress Report_ provides a qualitative assessment of the transmission mechanisms through which IE affects the quality of operations and examples of how this has worked in practice (Legovini 2011). Dhaliwal and Tulloch (2012) argue, based on the experience within J-PAL, that evidence can influence policy making, especially if it is independent, methodologically rigorous, substantive, actionable, easy to understand, cumulative, and easy to explain to constituents. On this last point, Masset et al. (2013) run an experiment in research dissemination to find that policy briefs on the impact of agricultural interventions did not have any effect in changing readers’ prior beliefs. Ruprah and Marcano (2009) show that technical assistance (that shares evidence with policy makers) has a positive impact on project outcomes and that this impact goes beyond the impact of financing alone. Overall, this literature is still nascent and inconclusive over the effect of IE on project implementation.

A complementary literature on the factors that explain the success of individual projects is also relatively thin. Some studies have investigated the country-level determinants of World Bank project outcomes. Explanatory factors include democracy, civil liberties, and conflict and different macroeconomic measures ranging from the quality of macroeconomic policy and market-oriented reforms to volatility and growth (Denizer et al. 2013). Other studies have looked at the link between project-level variations in outcomes and project-level factors, such as staff time devoted to preparation and supervision. Deininger, Squire, and Basu (1998) find that more analytical economic and sector work (as opposed to specific project preparation work) is correlated with the quality of World Bank loans and their level of disbursement. Kilby (2000, 2011, 2013a and 2013b) finds that longer project preparation is associated with better project ratings. Ika, Diallo, and Thuillier (2012) show a statistically significant and positive relationship between a set of five critical success factors (in particular, they focus on monitoring, coordination, design, training, and institutional environment) and project success. Denizer et al. (2013) find that 80 percent of total variation in project performance comes from _within_ -country variation, and correlates with project size, sector, quality of the project leader, and the amount of resources devoted to project preparation and supervision. We take this last study as a point of departure to motivate the importance of investigating whether IE research can affect the quality of projects.

5
# 3. INSTITUTIONAL BACKGROUND AND DATA

## 3.A. INSTITUTIONAL BACKGROUND

World Bank development financing is delivered in the form of projects, which are implemented by governments. Each project, led by a project leader,<sup>10</sup> follows the cycle of preparation, approval, effectiveness, implementation, and closing. During preparation and appraisal, the team designs the operation with a government agency, including planned activities and schedule of disbursements. During implementation the recipient government undertakes activities and submits disbursement requests to the Bank. The Bank project team supervises activities and clears disbursements for expenditures made in conformity with Bank guidelines (World Bank 2012a). These disbursements are recorded in the Bank’s systems. Because disbursements are made against agreed activities and supervised by the Bank, we use disbursements as a credible signal for project implementation. This is in line with common management practice: project disbursements are used as the main indicator of project success during implementation, and they are the main focus of weekly portfolio review meetings and fiscal year target exercises.

Furthermore, about two-thirds into the planned duration of a project, management conducts a midterm review to assess the progress of each project. Delays are common before midterm; this implies that schools, health facilities, or roads may not be built or staffed as planned. Furthermore, delayed projects may rush to make up for lost time when midterm approaches and in the relatively short time between midterm and project closing, when the remaining balance of the loan must be disbursed or is lost. We assume that implementing a project as planned and avoiding the rush to push out disbursements during the later stages of the project life cycle is bound to have beneficial effects on project outcomes. It might also have, on aggregate, important benefits for the reputation of an institution like the Bank, which receives donor funding and is periodically assessed against commitment and disbursement targets.

World Bank impact evaluations are conducted by the Bank’s internal researchers, often in collaboration with external researchers. These researchers use counterfactual analysis to evaluate policy interventions that are either Bank-financed or financed by other governments and institutions. This analysis only includes IEs of Bank-financed interventions, because they can be matched with project information from the Bank’s information systems. Most of these IEs are still ongoing, as are the projects they evaluate.

Notwithstanding the potential value of the knowledge generated for future projects, these IEs do operational research and have the potential for affecting current projects in real time through both the research and capacity-building channels. Research teams work with governments and project teams to affect policy design, agree on research design, adapt implementation modalities, and use

> 10 In the World Bank, project leaders are called team task leaders, or TTLs.

6
data and evidence generated to affect midcourse corrections in policy implementation as well as to influence scale-up or -down decisions.

These IEs are bundled with capacity-building activities and monitoring. Government and project teams attend workshops and trainings, are exposed to international evidence and the use of analytical tools for decision making, and participate in IE discussions and analysis. The IE research and field teams accompany the projects through their life cycle, monitoring IE implementation and data collection, and often supporting the implementation of project activities (World Bank 2006). Because of the bundled nature of Bank IEs, the analysis cannot separate the effects of IE research from that of capacity building and monitoring.

Furthermore, because the Bank IEs have been subject to internal quality assurance processes since 2007 (which means they require managerial approval, peer review, and discussion), they are less likely to be purely research led and more likely to have stronger links to policy. For these reasons, the estimates may not be applicable to more research-led IEs.

# 3.B. DATA

Our panel covers the period since 2005 when the Bank formally started doing IEs (Figure 1) and includes the 1,135 projects approved during 2005–11 that are in sectors that have at least one IE. Of these, 100 have an IE.

Project data are extracted from the World Bank’s Business Warehouse. Time-invariant characteristics include project id, region, country, sector, date of approval, and closing date. Timevarying characteristics are planned disbursements (initial and possible revisions), actual disbursements, and identity of the project leader. The data include 1,252 entries that are the universe of the projects approved by the Board between the first quarter of 2005 and the last quarter of 2011, across 28 quarters. As sectors with no IEs are excluded from the analysis, the sample drops to 1,135 projects, with up to those 28 quarterly observations, corresponding to 17,784 total quarterly observations.<sup>11</sup> Impact evaluation data comes from the IE database maintained by the Development Impact Evaluation Unit (DIME) in the Research Department of the World Bank. The database includes 377 entries in the period 2005–11. Of these, 100 are for Bank projects approved in the same period, or 9 percent of the project in our sample.<sup>12</sup> Time-invariant characteristics include IE id, region, country, sector, start date, and project id. Time-varying characteristics include IE status and identity of IE leader. We use project id to merge the data. When the variable “project id” is missing, it indicates that

> 11 The number of observations for a project depends on its vintage or year of approval.

> 12 The remaining is for policy interventions not financed by the World Bank, for which implementation data is not available.

7
the IE is evaluating interventions that are _not_ financed by a Bank project. Clearly, when project and IE data are merged, IEs with missing project id are automatically dropped.

Data is organized according to the quarter of implementation of the project. As such, quarter 1 is the first quarter of project implementation after effectiveness for all projects in the sample, and it can correspond to different calendar quarters depending on when the project started. By construction, there are 1,135 observations in quarter 1 with observations falling monotonically to 32 projects in quarter 28 (Figure 2). This represents an unbalanced panel of projects in different quarters, with an average project duration of 15 quarters across the 2005:Q1 to 2011:Q4 period. This sample has a substantial amount of variation in project vintage and duration across quarters.

We use the period before midterm as the institutional benchmark. As argued above, the period before midterm is of great importance as this is when projects are substantially off track. These delays may force teams to “push out” disbursements during the later phase of the project to avoid losing funding. We define the subset of projects that have reached midterm and the subset of quarters up to that midterm (an institutional benchmark) as the set that allows us to reduce composition effects. As Figure 3 illustrates, this approach reduces the heterogeneity in project duration and vintage, making sector composition more stable.

# 3.C. DESCRIPTIVE STATISTICS

Projects in our panel span 11 sectors and all six regions of the World Bank. The largest sectors in number of projects are agriculture and rural development, and transport, while the largest region is Africa. IEs are represented in regions and sectors to varying degrees (see Figure 4 and Table 1). Noninfrastructure sectors are well represented, while infrastructure sectors are less so. The public sector governance sector is excluded, as it did not have any IEs in this period of time.<sup>13</sup> In Table 2, we present descriptive statistics for projects with and without IE. The next few paragraphs discuss the statistics in this table. Eighty percent of the projects are still active (that is, under implementation) and the rest have closed, with different vintages (or start years) being evenly represented in the sample. About half of all projects (52 percent) have reached midterm, typically at two-thirds of their planned life.<sup>14</sup> The median midterm falls on quarter 13, with more than 50 percent of projects reaching midterm between Q11 and Q16. On average, projects are in their fourth year of implementation (15 quarters) out of the five years (20 quarters) originally planned. In terms of duration and other project characteristics, no difference between project with and without IE is found to be statistically significant, with the exception of loan size. The average loan size for projects without IE is $109 million versus $149 million for projects with IE.

Regarding the outcomes of interest, total disbursements are higher for projects with IE. This is in line with the larger loan size for projects with IE. Moreover, projects with IE disburse more relative to

> 13 This has changed since then.

> 14 Midterm should happen about halfway in project implementation. In practice, this is scheduled at two-thirds into project implementation due to delays.

8
loan size than projects without IE. Projects with IE that have actually reached midterm have disbursed 45 percent of the loan in total versus only 35 percent for projects without IE; and by closing date, they have disbursed 86 percent of the loan versus only 77 percent for projects without IE. Similarly, the average quarterly cumulative disbursement gap (the difference between planned and actual disbursements relative to total loan size) is lower for projects with IE (17 percent in the quarters up to midterm) than for projects without IE (27 percent). In other words, projects with IE are closer to their planned timeline by a large margin. Finally, relative to other projects in their sector, projects with IE disburse around 10 percent more by midterm as well as on average.<sup>15</sup> Because IEs start at different points in the project cycle, observing outcomes conditional on IE start date is instructive. A third of the IEs start by the first quarter, a third in the following year, and the rest thereafter (Figure 5). We plot the disbursements of (i) projects that do not have an IE (D=0), (ii) projects that will have an IE for quarters before the IE started (D=1 & T=0), and (iii) projects that have an IE and the IE has started (D=1 & T=1) to understand whether “IE assignment” (selection) or “doing IE” (treatment) is associated with increases in disbursements.

Over the project life cycle, data is consistent with the possibility that “doing IE” has a positive effect on disbursements. This is because, first, before starting an IE, projects without IE and projects that will have an IE in future quarters, (i) and (ii), have the same rate of disbursement (Figure 6). We interpret this as an indication of a good degree of comparability between projects with and without IE before actual selection into IE. Second, the rate of disbursement for projects that have started an IE, (iii), is consistently above (i) and (ii) for all quarters in the graph. This might suggest that disbursements increase _after_ the IE starts and thus more likely to be due to “doing IE” than to “IE assignment.” This is true for the whole sample as well as for the sample up to midterm (0). We discuss the selection issue in the next section.

In Figure 8 and 0 we look at the disbursement gap. We find that the disbursement gap is smaller for projects with IE, and especially before the IE starts. If we couple this with the facts above about actual disbursement (higher rate of disbursement for projects with IE), the observed behavior in disbursement gap means that projects that will start an IE tend to have more pessimistic implementation plans. For example, this could mean that project leaders were expecting these projects to be slow or difficult to implement, and may have selected into IE to get additional help. As such, data is suggestive that the measured effect could be a lower bound of the actual effect due to possible “adverse selection” of projects expected to be slower into IE. Indeed, adverse selection of potential slower projects into IE would tend to bias downward our estimates of the effect of IE on disbursements.

> 15 Often projects pay for the costs of data collection. These costs are in the order of $0.5 million or less than 1 percent of the loan. This can affect, but only marginally, project outlays. The costs of IE research, field coordination, and travel are borne by the Bank and are not part of this analysis.

9
When we then compare project disbursements relative to their sector average:<sup>16</sup> projects that will select into IE perform below their sector average before the IE starts and above the sector average after the IE starts (Figure 10 and Figure 11). This again points to the possibility of adverse project selection into IE that might bias our estimates downward.

Together, these observations suggest that implementation speeds up once projects start an IE, irrespective of the measure used. Further and by various measures, these projects do as well or worse before the IE starts than projects without IE. These findings suggest that our estimates may be a lower bound of the effect of IE on implementation.

# 4. ANALYTICAL STRATEGY

## 4.A. CONCEPTUAL FRAMEWORK

This section describes a simple conceptual framework used to estimate the effect of IE on project implementation. We model three dependent implementation variables: (y1) rate of disbursement (cumulative actual disbursement divided by total loan size); (y2) gap in the rate of disbursement (difference between cumulative planned and actual disbursement divided by total loan size); and (y3) deviation from sector-quarter average rate of disbursement (rate of cumulative disbursement divided by the sector average rate of cumulative disbursement in that quarter for all projects in the sample).

The variable y1 is used to measure the effect of the IE on the rate of implementation. Variable y2 is used to understand the effect of IE on implementation relative to the project team’s expectations. Finally, y3 is used to benchmark the effect of IE on implementation relative to the standard in the relevant sector.

We model implementation along the project life cycle as a function of time-specific factors, such as economic and political conditions; country-specific factors, such as institutional capacity; sectorspecific factors that slow down or speed up implementation, such as the construction of infrastructure; and project-specific factors, such as the quality of project management or supervision.

We model the effect of the IE as depending on (i) whether the project will have an IE, even if it has not started yet (project effect), and (ii) whether the IE started by that particular quarter (quarter effect). We will interpret (i) as the value of assignment to IE and (ii) as the effect of IE on implementation.

> 16 Sector averages are defined as the quarter averages of disbursement rates in each sector (that is, we have 28 different averages for the 11 sectors in our sample).

10
Because projects should eventually close their disbursement to commitment gap, the effect of the IE may not be linear. It might be stronger early on, as, perhaps, the IE provides for a larger team and more defined implementation plan, and taper off later.

# 4.B. PROJECT SELECTION INTO IE

Here we discuss project selection into IE, the main source of bias that may affect our empirical analysis. Project selection into IE is due to both institutional factors and project characteristics. Institutional factors determine bulk uptake into IE for different sectors at different points in time. This is because the World Bank made discrete efforts to increase IE in certain sectors and regions in certain years: education, health, and social protection since 2005; HIV/AIDS and malaria since 2007; agriculture and gender since 2008; finance and private sector since 2009; fragile states and financial literacy since 2010; infrastructure since 2011; business climate since 2012; and governance since 2013. Furthermore, commitments to donors in 2009 made IE compulsory for a sample of 2011–13 projects. These efforts resulted in discrete expansions of the IE portfolio in different sectors and regions.

In addition to these institutional factors, project-specific characteristics, such as the identity of the project leader, have been at play. If project leaders of potentially high-disbursing projects are more likely to select into IE, then projects with IE may disburse faster partially due to the project leader decision to adopt an IE. If, on the other hand, project leaders select into IE because their projects need additional help and support, then projects with IE will disburse at a slower pace. Descriptive statistics above provide some support for the latter case.

Our strategy to control for selection into IE relies on the assumption that the confounders are observed or time-invariant. Observable characteristics include portfolio-wide and project-specific factors. Portfolio factors include sector and country effects due to institutional efforts to introduce IE in those areas and related availability of funding for IE, and project leader fixed effects. In addition, we control for fiscal year and a third-order polynomial of average cumulative quarterly disbursement per sector. This function is added to capture potential composition effects that could confound the effect of IE. Project factors are accounted for via project fixed effects, and the identity of the project leader, or, alternatively, the recorded switches in project leader identity. The last variable is added because project leader turnover might slow down projects disbursements, as the new project leader will have to learn about the project and build relationships with the government counterpart.

Unobserved variables that we can control for include all characteristics related to projects’ quality that are time-invariant. There may be other unobservable characteristics that change over time that we cannot control for. For example, a project leader who selects into IE may change his/her attitudes toward IE as he/she learns the costs and benefits of doing IE. However, it is not clear how this may affect selection into IE prior to doing IE.

# 4.C. ESTIMATION STRATEGY

The estimation strategy approximates a difference-in-difference (DD) approach. The DD estimator identifies the average treatment effect of IE on the treated when the following assumptions are 11
satisfied: (i) in the absence of the treatment, the disbursements paths of projects with and without IE would evolve in parallel; and (ii) the unobserved characteristics that might affect selection into IE are additive and time-invariant. From the descriptive findings we discussed above, we have some indication that assumption (i) would be satisfied in our context. Indeed, we found that the disbursement trends of projects with IE before the IE starts are similar (at least for the disbursement rate measure) to those of projects without IE. Assumption (ii) is consistent with project team leaders having different attitudes toward IE, as long as those attitudes do not change over time. We use different methods and specifications to check robustness of the results.

Let _D_ be a dummy coded 0 for projects without IE and 1 for projects with IE, whether or not the IE started, and _T_ a dummy coded 0 in quarters before the IE starts and 1 thereafter. The DD estimate can be identified through the following pooled OLS model: where _Y_ is the outcome variable; _X_ a vector that includes dummies for number of times the project leader switches, the project leader’s Unique Personnel Number (UPI), and a third-order polynomial of average cumulative quarterly disbursement per sector; and _Z_ a vector that includes time-invariant variables such as the fiscal year, country, and sector. The vector of parameters γ captures the quarter effects. The composite error term ε _it_ is given by _ci_ + _uit_ , a time-invariant random component and an idiosyncratic component. The composite error term is assumed uncorrelated with observed variables and treatment status.<sup>17</sup> We can then interpret the parameter δ0 as measuring the effect of IE “assignment.” The parameter of interest is δ 1 , which measures the DD estimated effect of IE on outcome _Y_ .

The reason why our strategy “approximates” a DD approach is because some IEs start at the same time as the project; therefore, we do not have any _before_ data for this specific sample. Our DD estimator is therefore not the classic before-and-after framework. In particular, 3 percent of projects do not have a before comparison.<sup>18</sup> Note then that, unlike the standard DD model, where the coefficient of the dummy _D_ provides the difference in level of the outcome between projects with and without IE at baseline, here the coefficient δ0 gives the difference in level between the subset of projects that do not have an IE yet and the projects that will never have an IE.<sup>19</sup> This allows us to interpret the coefficient of _D_ as the

> 17 If the error term is uncorrelated with _X, D,_ and _T,_ then there is no need for the assumption (ii) above and the estimates will be consistent and efficient.

> 18 When we re-estimate our model excluding these observations, results do not change. Estimates available upon request.

> 19 As showed with the simple algebra, the effect of the 3 percent will be captured by the DD coefficient, whereas the coefficient of _D_ will capture the difference in level between projects that do not have an active IE yet and projects without IE in that quarter.

12
pre-IE treatment differences in outcome levels after the IE is already assigned. If δ0 were to be insignificantly different from zero, we would conclude that projects with and without IE have the same rate of disbursement before the IE starts, and that this would indicate comparability between IE and non-IE projects at the outset.

Our preferred estimation is a fixed effects (FE) model, because it allows for correlation between the error term and the covariates through the fixed effect<sup>_c_</sup> _i_ , the time-invariant component of the error term. This term captures characteristics of the projects that are correlated with the presence of an IE. With the FE transformation, the model takes the following form: i 1 i i i , (2) The transformation automatically drops all the variables that are fixed in time, such as dummies for ii ii YŸ = δδ TT̈ + XẌ ββ+ ZZ̈ λλ+ γγ + uü sector, fiscal year, and country and the _D_ dummy. The FE model generates consistent estimates when the unobserved fixed effects are correlated with the covariates. Given that projects approved within the same sector in a given time are more likely to share similar characteristics, standard errors are clustered at sector-time level.

To check for robustness, we also estimate a pooled OLS, and a random effects (RE) model (estimated through a feasible generalized least squares model). These models require fewer restrictions because they do not control for project fixed effects. The reason for reestimating (1) using RE is to improve efficiency and to understand how important project fixed effects are (through the comparison of RE and FE). Given the relatively long horizon in our panel, the error term will be serially correlated across time. We try to control for this clustering standard errors at the sector and quarter levels.

13
<!-- Start of picture text --> Main Results 49.1%5***<br>40.8%*** (in percentage40.0%*** of outcome mean)<br>15.6%<br>Rate of Gap rm) Rate of Rate of G Il) Rate of<br>Disbursement (to Disbursement - Disbursement Disbursement -<br>midterm) Adjusted by Sector (overall) Adjusted by Sector<br>Average (to Average (overall)<br>midterm)<br>-45.3%**<br>-54.2%***<br><!-- End of picture text -->
percent level) and relative to sector disbursements (49 percent, significant at 1 percent level), although not for average cumulative disbursements. Smaller point estimates are due to the partial convergence of disbursements to total loan amount as projects approach their closing date. This is only partial because (i) most projects in our sample have not yet closed (74 percent), and (ii) projects on average do not disburse their whole loan amount (78 percent in our sample). Average cumulative disbursements estimates are also affected by the substantial amount of heterogeneity in duration that is introduced when the whole sample is used, while the heterogeneity is controlled when the same disbursements are benchmarked against project plans or sector averages. The effects are economically large even using these smaller point estimates.

The summary estimates above are based on an FE model that includes as controls dummies for the number of times the project leader changes and a third-order polynomial of average cumulative quarterly disbursement per sector. The estimates are robust to different specifications and the inclusion of different sets of controls. We briefly discuss these additional estimates below.

# 5.A. QUARTERS UP TO MIDTERM

The effect of IE on disbursements for the quarters up to midterm is large, positive, and significant. It is also robust to different specifications (Table 4). In the FE model, the estimates range between 3.5 and 6 percentage points, corresponding to 24 to 41 percent of cumulative disbursements, significant at 1–5 percent level (with the exceptions of the specification in the last column of Table 4). The average project in the sample has only disbursed 14.7 percent of its loan amount in the average quarter up to midterm, or only about half of what was originally planned (29 percent). The effect size is thus large in absolute terms, contributing substantially to project implementation. Furthermore, the estimates are stable across different models, ranging between 5.6 and 6.4 percentage points (pps) in the RE and 5.6 and 8.2 pps in the OLS (all significant at 1–10 percent level). The latter models also provide an estimate for the coefficient on the IE assignment. Point estimates of the effect of the dummy _D_ are negative, suggesting a degree of adverse selection into IE. We conclude that the IE seems to have a very considerable effect on the implementation speed up to midterm and that this effect is not due to IE assignment (selection) but to doing IE (treatment).

Even more striking is the result on the effect of doing IE on the gap between planned and actual cumulative disbursements during the first two-thirds of project implementation. This result incorporates the acceleration of disbursements described above, but now takes into account the expectations that the project team originally had on the speed of implementation of their specific project. As can be seen in Table 5, the FE estimated effect of IE on the planned to actual gap ranges between -5 and -8 pps (significant at 1 percent level), which, relative to the mean gap of 14.4 percent of the loan amount, implies a 35 to 54 percent reduction in the disbursement gap. The effect is economically important: projects with IE reduce delays by half. Results carry through using alternative models, and the effect of IE assignment is not significantly different from zero when project leader fixed effects are included (with the exception of few cases in the OLS specification).

Because projects in different sectors may follow different implementation paths (for example, infrastructure projects are slower to implement than non-infrastructure projects), we worry about 15
controlling for sector effects along the project life cycle. We thus estimate the average disbursement paths in each sector and normalize specific project disbursements to it.<sup>20</sup> When we then re-estimate the model, we find that the IE pushes disbursements way above the norm achieved by other projects in that same sector. Table 6 shows the estimates for the average effect of the IE on the rate of disbursement adjusted by the sector quarter average. By midterm, the FE estimates of the effect of IE range between 40 and 42 percent increase above the norm. The effects are highly significant in both statistical (1–5 percent level) and economic terms. Once OLS and RE models are compared, point estimates are consistent, and, if anything, larger. In those models, some of the variation is captured by the _D_ dummy, with a zero or negative coefficient, giving limited support to the possibility that there might be adverse selection of slower projects into IE as we argued above.

# 5.B. ALL QUARTERS

Around and after midterm, the projects that are delayed might rush to close their disbursement gap in view of the approaching review and closing date. Therefore, we would expect the effect of the IE over the whole implementation period to be smaller than for the period before midterm. Indeed, the FE point estimates for cumulative disbursements when using quarters up to closing date are only about a third of the effect to midterm and are not statistically significant (Table 7). It is hard, however, to interpret the results for the whole sample straightforwardly because a substantial amount of heterogeneity in duration is introduced. Specifically, there are very few projects with active IE after quarter 20. Therefore, using the whole sample of data assigns too much weight to quarters where the effect of IE is expected to be zero and IEs are almost absent.

A different story arises, however, when we benchmark disbursements against project plans or sector averages. Here the IE continues to have strong and statistically significant effects, with FE estimates ranging between 3.4 and 6.6 pps, or 23 to 45 percent of total gap. The effect of the IE on disbursements relative to average sector disbursements is also large and significant (40 to 42 percent). The results are relatively stable across models and specifications and statistically significant at 1–10 percent level (Table 8 and Table 9).

The difference between the results on disbursements and disbursement gap is instructive. The analysis on disbursements makes no presumption on the desired path of disbursements and finds little IE effects. When, however, the analysis incorporates project-specific expectations on their disbursement path, the IE has large and significant estimated effects. An interpretation of this is that planned disbursements might capture project’s implementation path in more subtle ways than project fixed effects. This seems to be especially important for the analysis across all quarters, as there is a lot of heterogeneity in project duration. We conclude that, given the unbalanced panel, planned expenditure helps control for the composition effect across quarters and helps identify the IE effects. The same argument carries through in the case of sector-specific disbursement paths.

> 20 By construction, for the average project, the ratio between rate of disbursement and average disbursement is 1.

16
Overall, we find positive and significant effects of IE on project disbursements, especially when measured against plans and sector averages. The effects are stable across models and specifications, especially up to midterm, where comparability is greater and issues of composition are less important. For the quarters up to midterm, the effect is so large as to reduce the breach between project plans and actual implementation by half.

# 5.C. POTENTIAL MECHANISMS

The potential channels through which World Bank IE research can affect projects are described through anecdotal evidence in Legovini (2010, 2011). We also build on Legovini and Bedoya (2014), who develop a set of policy influence indicators for IEs that cover policy design, capacity building and data, policy implementation, and evidence-based decisions. They collect data from IE teams for a set of 116 IEs covering both World Bank projects and other investments. We report here some findings from that study with the aim of presenting the channels that could be more relevant to explain the effect of IE.

During the early stages of engagement between research and project teams, the issue of rigor in policy design is made more salient. Teams report that existing evidence is incorporated in policy design (42 percent of the cases) and a structured learning strategy is introduced. The latter is measured by counting whether projects introduce experimental treatment arms to compare different implementation models or mechanisms (53 percent of the cases versus none for projects without IE).

As part of this process, teams develop detailed implementation plans and become more aware of the potential contribution of their work to the development objectives of their countries and their role in securing results. Team motivation may be affected by the process of making plans more actionable and the link with results more explicit.

During preparation and implementation, several efforts are geared toward increasing teams’ capacity. We monitor client attendance to one-week IE workshops<sup>21</sup> (88 percent) or data training (36 percent), and incidence of discussions on baseline results (74 percent) and IE results (75 percent). Capacity is measured by participant learning and the technical quality of the planned IEs. As measured by the difference in IE workshop pre- and post-test results, participants increase their impact evaluation knowledge by one standard deviation on average. Further, as measured by doubleblind external reviews, the average technical rating for proposals submitted by teams who participated in an IE workshop exceeds by half a standard deviation that of proposals submitted by other teams.<sup>22</sup> In addition to training, capacity is affected by supplementing the operations teams with a research and by placing a full-time field coordinator (100 percent).

> 21 See events at dime.worldbank.org

> 22 Authors’ calculations.

17
Further effort is expanded on collection of high-quality data, often the first set of sector data an agency will have access to. IE-generated data provide a client with at least two rounds of survey data (84 percent) and support to help improve administrative data systems (48 percent). Survey data are generally not available for projects without IE.

Evidence-based decisions are taken at multiple stages of IE implementation. Policy changes are recorded following baseline results discussions (36 percent), and when different mechanisms are tested (23 percent). Policy scale-up and -down decisions are recorded following discussions on IE results in 30 percent of the cases.

The relative importance of these different mechanisms will be the subject of future research as we expand the IE coverage and frequency of data collection of these monitoring indicators.

# 6. CONCLUSIONS

This is the first study that investigates the effect of IE research on contemporaneous measures of aidfinanced project implementation. It builds on the notion that the effectiveness of aid greatly depends on the quality of projects through which aid is delivered, and builds a case for a fruitful complementarity between aid financing and IE research. The study presents evidence that impact evaluation may help improve project implementation. Since project implementation is a necessary condition (although not sufficient) for projects to reach their development outcomes, then impact evaluation might be an enabling mechanism to secure developmental effectiveness.

We find that impact evaluation has large and significant effects on the disbursements of World Bank projects, increasing the average level of quarterly disbursements by 40 percent by midterm, helping projects reduce delays relative to plan by half, and increasing by 40 percent project disbursements relative to the average in the typical project in the same sector.

We conclude that projects with IE are more likely to implement their activities as planned and, in so doing, are more likely to achieve their objectives. This is of significance for both the Bank and development policy. For the Bank, the introduction of IE might help avert the need to push disbursements out toward the latter half of project duration, or to restructure and cancel nonperforming projects. It can also help manage the potential reputational risk of committing funding without being able to secure their disbursements. More important, for developing countries, avoiding delays may mean that schools, health facilities, or roads are built now instead of next year or not at all, securing the potential for educating one more generation, saving more lives, or connecting to market more people and goods.

Furthermore, our results support the idea that project financing and IE research are complementary outputs of the World Bank. This has implications for the use of IE as an operational tool to improve 18
delivery of projects, and for weighing the costs of IE research<sup>23</sup> against the benefits on the Bank’s project portfolio. For development organizations like the World Bank, understanding the degree of substitutability or complementarity between these outputs can help strike an optimal balance between project financing and IE research to help us reach the efficient frontier.

A more general implication of this work is that the development community overall may be underinvesting in the type of research that has positive externalities on development investments, thus forsaking the opportunity to increase developmental effectiveness. Presently, IE research coverage is about 10 percent of the World Bank portfolio. This is likely to be lower in other development banks and institutions that, on average, have less research capacity. Whether this level strikes the optimal balance remains to be assessed.

Finally, our results question the notion that impact evaluation is solely a knowledge-generating public good, and puts a value to the use of impact evaluation as a private good, that is, one that helps individual projects deliver on their promises. If IE has public and private good features, then the case for what constitutes knowledge generation value and who should contribute to IE financing could be investigated.

Future research can help better inform this debate. First, research should focus on understanding the underlying mechanisms through which IE research affects the path of project implementation and tracing the pathways through which these effects can be strengthened. Second, research should focus on quantifying the benefits of timely project implementation on development outcomes and identifying whom the benefits accrue to. This could help inform allocative choices in institutions like the World Bank and other development agencies.

# REFERENCES

Bruhn, M., L. Leao, A. Legovini, R. Marchetti, and B. Zia. 2013. “The Impact of High School Financial Education: Experimental Evidence from Brazil.” Policy Research Working Paper 6723, World Bank, Washington, DC.

Brune, L., X. Giné, J. Goldberg, and D. Yang. 2011. “Commitments to Save: A Field Experiment in Rural Malawi.” Policy Research Working Paper 5748, World Bank, Washington, DC.

Denizer, C., D. Kaufmann, and A. Kraay. 2013. “Good Countries or Good Projects? Macro and Micro Correlates of World Bank Project Performance.” _Journal of Development Economics_ 105:288–302.

> 23 The cost to the Bank of a completed IE product was, on average, $279,493 in fiscal 2007–14. This represents less than 0.2 percent of loan size.

19
Deininger, K., L. Squire, and S. Basu. 1998. “Does Economic Analysis Improve the Quality of Foreign Assistance?” _World Bank Economic Review_ 12 (3): 385–418.

Dhaliwal, I., and C. Tulloch. 2012. “From Research to Policy: Using Evidence from Impact Evaluations to Inform Development Policy.” _Journal of Development Effectiveness_ 4 (4): 515–36.

Friedman, J. M., M. Vledder, M. Sjoblom, and P. Yadav. Forthcoming. “Improving Patient Access to Malaria and Other Essential Medicines in Zambia: Results of a Pilot Project.” Presented at the Transport Learning Event, Washington, DC, 2010.

http://siteresources.worldbank.org/INTTRANSPORT/Resources/3362911297096897336/7715763-1297096948414/Presentation-Friedman.pdf.

Ika, L. A., A. Diallo, and D. Thuillier. 2012. “Critical Success Factors for World Bank Projects: An Empirical Investigation.” _International Journal of Project Management_ 30 (1): 105–16.

Kilby, C. 2000. “Supervision and Performance: The Case of World Bank Projects.” _Journal of Development Economics_ 62 (1): 233–59.

Kilby, C. 2011. “Assessing the Contribution of Donor Agencies to Aid Effectiveness: The Impact of World Bank Preparation on Project Outcomes.” Villanova School of Business Economics Working Paper 20, Villanova University, Pennsylvania.

Kilby, C. 2013a. “The Political Economy of Project Preparation: An Empirical Analysis of World Bank Projects.” _Journal of Development Economics_ 105:211–25.

Kilby, C. 2013b. “An Empirical Assessment of Informal Influence in the World Bank.” _Economic Development and Cultural Change_ 61 (2): 431–64.

Kondylis, F., M. Jones, and D. Stein. 2014. “Commitment, Mental Accounting, & Agricultural Investment: Evidence from Rwanda.” Mimeo. World Bank Legovini, A. 2010. “Development Impact Evaluation Initiative: A World Bank–Wide Strategic Approach to Enhance Developmental Effectiveness.” World Bank, Washington, DC. http://siteresources.worldbank.org/INTDEVIMPEVAINI/Resources/39981991286546178578/7465778-1291306572028/Legovini_dime_paper_ext.pdf.

Legovini, A. 2011. “DIME Progress Report.”, World Bank, Washington, DC. http://siteresources.worldbank.org/INTDEVIMPEVAINI/Resources/39981991372170088287/DIME_progress_report_2012.pdf.

Legovini, A., and G. Bedoya. 2014. “Dime’s Current Portfolio and Approach to Impact Evaluation.” Technical note, World Bank, Washington, DC.

20
Masset, E., M. Gaarder, P. Beynon, and C. Chapoy. 2013. “What Is the Impact of a Policy Brief? Results of an Experiment in Research Dissemination.” _Journal of Development Effectiveness_ 5 (1): 50–63.

Millennium Challenge Corporation (MCC). 2012. “MCC’s Independent Evaluations.” Fact sheet, MCC, Washington, DC. http://www.mcc.gov/documents/press/factsheet-2012002116501-independentevaluations.pdf.

Neilson, S. 2001. “Knowledge Utilization and Public Policy Processes: A Literature Review.” Ottawa: IDRC Evaluation Unit.

Ruprah, I., and L. Marcano. 2009. “Does Technical Assistance Matter? An Impact Evaluation Approach to Estimate Its Value Added.” _Journal of Development Effectiveness_ 1 (4): 507–28.

Temple, J. R. 2010. “Aid and Conditionality.” In _Handbook of Development Economics,_ edited by D. Rodrik and M. Rosenzweig, 5:4415–523. Amsterdam: North Holland.

Thomas, V., and J. Tominaga. 2010. “Evaluation for Better Development Results.” _Journal of Development Effectiveness_ 2 (3): 371–86.

World Bank. 2006. “Impact Evaluation and the Project Cycle: Doing Impact Evaluation No. 1.” http://siteresources.worldbank.org/INTISPMA/Resources/Training-Events-andMaterials/ie_and_projectcycle.pdf.

World Bank. 2009. _Annual Review of Development Effectiveness, 2009: Achieving Sustainable Development_ . Washington, DC: IEG, World Bank.

http://siteresources.worldbank.org/EXT2009ARDE/Resources/arde09_web.pdf.

World Bank. 2012a. _The World Bank Annual Report 2012: Volume 3. Annexes._ Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/11846.

World Bank. 2012b _World Bank Group Impact Evaluations. Relevance and Effectiveness.._ Washington, DC: IEG, World Bank. http://ieg.worldbank.org/Data/reports/impact_eval_report.pdf World Bank. 2014a. “World Bank Corporate Scorecard, April 2014.” Brochure, World Bank, Washington, DC.

http://siteresources.worldbank.org/CSCARDEXT/Resources/2014_WBG_corporate_scorecard_eversion.pdf.

World Bank. 2014b. “Review of the World Bank FY13 Portfolio and Update on Actions to Assure Quality.” World Bank, forthcoming

# **FIGURES AND TABLES**

21
<!-- Start of picture text --> AP SY SY oP oh OF © CP FP MV GF SF FS<br>DEST ST NT NT NT NT NI BP a a PP<br><!-- End of picture text --> <!-- Start of picture text --> 1200 12%<br>3945 61<br>1000 - 10%<br>70<br>71<br>800 64 —— —— 8%<br>57<br>} ama 57 Wi "<br>600 / | | / 49 Projects with IE<br>46 As, V mmm Projects without IE<br>y UY a 49 6%<br>% of projects with IE<br>“| | 36 ——<br>400 34<br>V, 4 0 28 4%<br>18<br>16<br>0 0%<br>123 45 67 8 9 101112 1314151617 1819 2021 22 2324 252627 28<br><!-- End of picture text -->
<!-- Start of picture text --> Quarters<br>1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28<br>9 ————————————<br>80 |<br>1!E_—_ EE ——_—_—_—_—_—_—_—_—~<br>273<br>P 385 eee<br>r 475 OT 592 projects<br>les HH Feached<br>eJ 65S ~=m midterm<br>l<br>c 780 I<br>t 841 ]<br>s 948 '<br>1,005 Average<br>1,113 duration up to<br>1,134 midterm is<br>13.37 quarters<br><!-- End of picture text -->

<!-- Start of picture text --> 316<br>225<br>210 209<br>185<br>147 152<br>126<br>114 113<br>107 100<br>82<br>65 63<br>47<br>34<br>24 17 2 2 11 12 22 4 16 3 5 4 8 4 24 2 15<br>SS CSF ESE S SE S € Xf F SF<br>oeFy’CFS» CONE€ £F S&SEN. EFET aCES EESENS) TPS LM SH<br>©eS < FSOs ohOr SX® oSee « eSe<br>Ke @ © & & ~ & Sy<br>s & eSSS 9SF F ysSF<br>s ew Ss<br>sv Foe<br>~~ ey<br>$ ® 6<br>ye oeRS<br>s<br>Se<br>BAIlprojs @withlE<br><!-- End of picture text -->
<!-- Start of picture text --> 40 37<br>35<br>30<br>25<br>20<br>15<br>10 9 8 8<br>5 6 5 4 4<br>3 3 2 1 2 1 2 1<br>0<br>1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16<br>m Number of projs with IE at<br><!-- End of picture text --> <!-- Start of picture text --> 1<br>0.9<br>-OY<br>0.7 —e .<br>e° °<br>0.6 a<br>e<br>e<br>0.5 .<br>e<br>: e eeeeD-0<br>- YY e<br>- fe<br>e<br> gs ~<br>0.10 ruZS tt e<br>123 45 67 8 9 101112 131415 16 17 18 19 20 21 22 23 24 25 2627 28<br><!-- End of picture text -->
<!-- Start of picture text --> 0.6<br>0.5 ~<br>e<br>e<br>e<br>e<br>0.4 — e<br>°<br>wee ee eo<br>° em D=1 & T==0<br>0.3<br>. ° cmeD=1 & T==1<br>ee eeeeD=0<br>e<br>0.2 9<br>e<br>le<br>0.10 Via<br>123 45 67 8 9 10111213141516 1718 19 20 21 22 23 24 25 2627 28<br>0.35<br>0.3 °° °<br>°° e<br>-<br>0.25 . “s<br>° om<br>e ee,<br>0.2 s<br>e<br>e<br>e e<br>0.15 nd —<—\ r—_\ ° °<br>0.1 — IN—_ COi ° e emmeesee eD 0D = 1 & T==1<br>0 = tw<br>123 45 67 8 9 1011121314151617 18 19 20 21 22 23 24252627 28<br>--0.1SY<br>-0.15<br><!-- End of picture text -->
<!-- Start of picture text --> 0.5<br>°<br>ee<br>ee eeee<br>0.4 °<br>° e<br>ete e<br>°<br>ePod e ee<br>0.3<br>e °<br>e °<br>i ° °e ==D=1 & T==0<br>0.2 ° \N ‘<br>J} commeD=1 & T==1<br>— eeee D=0<br>0.10 ; ~ /<br>2345 67 8 9 1011121314151617 1819 20 21 22 23 24 25 26 27 28<br>-0.1<br>2.5<br>1.5<br>emD=1 & T==0<br>NY Peweeee 118100<br>05 |<br>0<br>123 45 67 8 9 101112 1314 15 16 17 18 19 20 21 22 23 24 25 2627 28<br><!-- End of picture text -->
<!-- Start of picture text --> 4<br>3.5 |<br>. |<br>2.52 | eum D=1 & T==0<br>eeeeD=-0<br>:: .\{~~TO Sr Lf A<br>ee ie<br>0.50 /<br>123 45 67 8 9 101112 131415 16 1718 19 20 21 22 23 24 25 2627 28<br><!-- End of picture text -->
# Table 1. PROJECTS WITH AND WITHOUT IE BY SECTOR AND FISCAL YEAR

|**_Fiscal year_**|**20**|**05**|**2**|**006**|**2**|**007**|**2**|**008**|**20**|**09**|**20**|**10**|**2011**|**TO**|**TAL**|**IE/TOTAL**|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|**_Sector_**|**IE**|**No IE**|**IE**|**No IE**|**IE**|**No IE**|**IE**|**No IE**|**IE**|**No IE**|**IE**|**No IE**|**IE**<br>**No IE**|**IE**|**No IE**|**(%)**|
|Agriculture and Rural Development|0|32|4|42|4|35|3|30|7|28|5|25|1<br>9|24|201|12|
|Education|3|22|3|17|5|15|1|16|3|15|2|10|0<br>2|17|97|18|
|Energy and Mining|0|21|2|25|0|12|0|26|0|20|0|14|0<br>6|2|124|2|
|Environment|0|9|1|9|0|2|1|4|0|4|0|3|0<br>1|2|32|6|
|Financial and Private Sector Development|2|17|0|14|2|9|2|6|2|10|2|12|1<br>3|11|71|15|
|Health, Nutrition and Population|1|20|4|16|3|13|2|18|1|13|1|13|0<br>2|12|95|13|
|Social Development|2|4|0|4|0|3|1|0|1|3|0|3|0<br>1|4|18|22|
|Social Protection|1|10|4|3|1|2|3|9|6|8|0|16|1<br>1|16|49|33|
|Transport|0|17|1|22|1|29|1|24|0|20|0|30|0<br>2|3|144|2|
|Urban Development|1|18|2|13|0|18|0|21|1|10|0|21|1<br>7|5|108|5|
|Water|1|17|0|12|1|22|0|16|1|15|1|14|0<br>0|4|96|4|
|Total|11|187|21|177|17|160|14|170|22|146|11|161|4<br>34|100|1035|10|

28
Table 2. DESCRIPTIVE STATISTICS

||**No IE**|**IE**|**Total**|**Sample size**<br>**P**|**value from T test**|
|---|---|---|---|---|---|
|**_Project Characteristics_**||||||
|_Number of projects_|1035|100|1135|||
|_Number of quarters_|16540|1145|17685|||
|_Fiscal Year of Effectiveness is 2005_|187|11|198|||
|_... 2006_|177|21|198|||
|_... 2007_|160|17|177|||
|_... 2008_|170|14|184|||
|_... 2009_|146|22|168|||
|_... 2010_|161|11|172|||
|_... 2011_|34|4|38|||
|_Activeprojects_|760|78|838|||
|_Closedprojects_|275|22|297|||
|_Total loan size committed(USD million)_|109|149|112||0.07|
|_Midterm review completed_|537|55|592|||
|_Number of TTLsperproject_|2.1|2.11|2.1||0.92|
|**_Duration_**||||||
|_Actual duration inquarters(all)_|15.69|15.39|15.67|1135|0.68|
|_Planned duration inquarters(all)_|20.02|19.64|19.99|1135|0.43|
|_Actual duration inquarters(activeprojects)_|13.87|14.30|13.91|838|0.59|
|_Planned duration inquarters(activeprojects)_|21.01|20.65|20.97|838|0.47|
|_Actual duration inquarters(closedprojects)_|20.72|19.22|20.61|297|0.20|
|_Planned duration inquarters(closedprojects)_|17.29|16.04|17.19|297|0.24|
|_Actual duration(projects with midterm review)_|18.11|18.52|18.15|592|0.60|
|_Actual Duration up to midterm review_|13.38|13.25|13.37|592|0.79|
|**_Disbursement_**||||||
|**_Planned disbursement_**||||||
|_Average (USD million)_|61.13|86.45|63.36|1135|0.04|
|_Average at midterm(USD million)_|52.86|76.66|55.07|592|0.05|
|_Rate(totalplanned disb/loan committed)_|0.64|0.60|0.63|1135|0.34|
|_Rate at midterm(if midterm reached)_|0.62|0.62|0.62|592|0.9|
|**_Actual disbursement_**||||||
|_Average (USD million)_|44.23|79.20|47.31|1135|0.00|
|_Average at midterm(USD million)_|34.93|54.85|36.78|592|0.07|
|_Rate(total disbursed/loan committed)_|0.44|0.49|0.44|1135|0.17|
|_Rate at midterm_|0.35|0.45|0.36|592|0.00|
|_Rate atquarter 4_|0.06|0.07|0.06|1116|0.36|
|_Rate atquarter 8_|0.15|0.21|0.16|953|0.00|
|_Rate atquarter 12_|0.28|0.36|0.29|785|0.00|
|_Rate atquarter 16_|0.43|0.50|0.44|589|0.07|
|_Rate atquarter 20_|0.57|0.60|0.57|391|0.54|
|_Rate at closing (closedprojects)_|0.77|0.86|0.78|297|0.16|
|**_Disbursementgap (planned - actual disbursement)_**||||||
|_Average perquarter_|0.01|0.005|0.009|17685|0.00|
|_Average_|0.15|0.07|0.14|1135|0.05|
|_Average at midterm_|0.27|0.17|0.26|592|0.01|
|_Average at closing(closedprojects)_|0.00|-0.18|-0.01|297|0.10|
|**_Actual disbursement deviationfrom sector/quarter average _**||||||
|_Rate_|0.99|1.08|0.99|17629|0.25|
|_Rate at midterm_|0.89|1.09|0.91|7867|0.10|
|**_Time tofirst disbursement(quarters)_**|4.15|3.91|4.13|1126|0.23|

29
– Table 3. SUMMARY TABLE FIXED EFFECTS ESTIMATES

||**Rate of**<br>**Disbursement -**<br>**Up to Mid-Term**|**Gap - Up to Mid-**<br>**Term**|**Rate of**<br>**Disbursement -**<br>**Adjusted by**<br>**Sector Average**<br>**– Up to Mid-**<br>**Term**|**Rate of**<br>**Disbursement**|**Gap**|**Rate of**<br>**Disbursement -**<br>**Adjusted by**<br>**Sector Average**|
|---|---|---|---|---|---|---|
|**T (quarters with IE==1)**|0.060***|-0.078***|0.40***|0.038|-0.067**|0.45***|
||(3.00)|(-3.08)|(3.39)|(1.54)|(-2.47)|(3.15)|
|**_TTL switch dummies_**|Y|Y|Y|Y|Y|Y|
|**_Polynomial of sector averages_**|Y|Y|Y|Y|Y|Y|
|**_Observations_**|7918|7921|7889|17784|17784|17728|
|**_Outcome mean_**|0.147|0.144|1.00|0.244|0.148|0.917|
|**_Effect in %_**|40.8%|54.2%|40.0%|15.6%|45.3%|49.1%|

_Note:_ Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle.

Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 30
# Table 4. EFFECT OF IE ON RATE OF DISBURSEMENT UP TO MIDTERM

|||DID|-OLS|||DID|-RE|||DI|D-FE||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|**T (quarters with IE==1)**|0.082***|0.056***|0.082***|0.056***|0.063***|0.056***|0.064***|0.056***|0.060***|0.036*|0.060***|0.035|
||(10.0)|(4.59)|(9.99)|(4.56)|(3.43)|(3.16)|(3.47)|(3.15)|(2.99)|(1.69)|(3.00)|(1.64)|
|**D (project with IE==1)**|-0.030***|-0.052***|-0.030***|-0.052***|-0.015|-0.052**|-0.015|-0.052**|||||
||(-5.31)|(-4.17)|(-5.32)|(-4.16)|(-0.96)|(-2.52)|(-0.96)|(-2.51)|||||
|**_TTL switch dummies_**|Y|N|Y|N|Y|N|Y|N|Y|N|Y|N|
|**_TTL UPI dummies_**|N|Y|N|Y|N|Y|N|Y|N|Y|N|Y|
|**_Polynomial of sector averages_**|N|N|Y|Y|N|N|Y|Y|N|N|Y|Y|
|**_Observations_**|7918|7918|7918|7918|7918|7918|7918|7918|7918|7918|7918|7918|
|**_Outcome mean_**|0.147|0.147|0.147|0.147|0.147|0.147|0.147|0.147|0.147|0.147|0.147|0.147|
|**_Effect in %_**|55.8%|38.1%|55.8%|38.1%|42.9%|38.1%|43.5%|38.1%|40.8%|24.5%|40.8%|23.8%|

_Note:_ Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle, sector, fiscal year when project became effective, and country.

Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 31
# Table 5. EFFECT OF IE ON GAP UP TO MIDTERM

|||DID|-OLS|||DI|D-RE|||DID|-FE||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|**T (quarters with IE==1)**|-0.056***|-0.045***|-0.056***|-0.045***|-0.073***|-0.045|-0.073***|-0.045|-0.078***|-0.051*|-0.078***|-0.051*|
||(-4.86)|(-2.85)|(-4.83)|(-2.86)|(-3.15)|(-1.62)|(-3.16)|(-1.62)|(-3.07)|(-1.79)|(-3.08)|(-1.78)|
|**D (project with IE==1)**|-0.0069|0.035**|-0.0069|0.035**|0.0081|0.035|0.0081|0.035|||||
||(-0.83)|(2.09)|(-0.83)|(2.07)|(0.43)|(1.34)|(0.44)|(1.33)|||||
|**_TTL switch dummies_**|Y|N|Y|N|Y|N|Y|N|Y|N|Y|N|
|**_TTL UPI dummies_**|N|Y|N|Y|N|Y|N|Y|N|Y|N|Y|
|**_Polynomial of sector averages_**|N|N|Y|Y|N|N|Y|Y|N|N|Y|Y|
|**_Observations_**|7918|7918|7918|7918|7918|7918|7918|7920|7918|7918|7921|7918|
|**_Outcome mean_**|0.144|0.144|0.144|0.144|0.144|0.144|0.144|0.144|0.144|0.144|0.144|0.144|
|**_Effect in %_**|38.9%|31.3%|38.9%|31.3%|50.7%|31.3%|50.7%|31.3%|54.2%|35.4%|54.2%|35.4%|

_Note:_ Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle, sector, fiscal year when project became effective, and country.

Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 32
# Table 6. EFFECT OF IE ON RATE OF DISBURSEMENT ADJUSTED BY SECTOR AVERAGE UP TO MIDTERM

|||DID|-OLS|||DID|-RE|||DID|-FE||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|**T (quarters with IE==1)**|0.74***|0.48***|0.74***|0.48***|0.46***|0.40***|0.44***|0.40***|0.42***|0.41***|0.40***|0.40***|
||(5.60)|(4.03)|(5.54)|(4.00)|(3.88)|(3.03)|(3.68)|(2.93)|(3.62)|(3.04)|(3.39)|(2.91)|
|**D (project with IE==1)**|-0.46***|-0.21|-0.46***|-0.21|-0.30|-0.21|-0.29|-0.21|||||
||(-3.84)|(-1.53)|(-3.81)|(-1.53)|(-1.44)|(-1.02)|(-1.38)|(-0.99)|||||
|**_TTL switch dummies_**|Y|N|Y|N|Y|N|Y|N|Y|N|Y|N|
|**_TTL UPI dummies_**|N|Y|N|Y|N|Y|N|Y|N|Y|N|Y|
|**_Polynomial of sector averages_**|N|N|Y|Y|N|N|Y|Y|N|N|Y|Y|
|**_Observations_**|17784|17784|17784|17784|17784|17784|17784|17784|17784|17784|17784|17784|
|**_Outcome mean_**|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|
|**_Effect in %_**|74.0%|48.0%|74.0%|48.0%|46.0%|40.0%|44.0%|40.0%|42.0%|41.0%|40.0%|40.0%|

_Note:_ Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle, sector, fiscal year when project became effective, and country.

Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 33
# Table 7. EFFECT OF IE ON RATE OF DISBURSEMENT

|||DID|-OLS|||DI|D-RE|||DID|-FE||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|**T (quarters with IE==1)**|0.082***|0.037***|0.083***|0.037***|0.039*|0.011|0.040*|0.011|0.037|0.0091|0.038|0.0096|
||(9.21)|(3.71)|(9.27)|(3.75)|(1.69)|(0.54)|(1.73)|(0.56)|(1.50)|(0.44)|(1.54)|(0.46)|
|**D (project with IE==1)**|-0.029***|0.0044|-0.030***|0.0044|-0.0035|0.011|-0.0044|0.011|||||
||(-4.66)|(0.44)|(-4.70)|(0.43)|(-0.20)|(0.45)|(-0.25)|(0.45)|||||
|**_TTL switch dummies_**|Y|N|Y|N|Y|N|Y|N|Y|N|Y|N|
|**_TTL UPI dummies_**|N|Y|N|Y|N|Y|N|Y|N|Y|N|Y|
|**_Polynomial of sector averages_**|N|N|Y|Y|N|N|Y|Y|N|N|Y|Y|
|**_Observations_**|17784|17784|17784|17784|17784|17784|17784|17784|17784|17784|17784|17784|
|**_Outcome mean_**|0.244|0.244|0.244|0.244|0.244|0.244|0.244|0.244|0.244|0.244|0.244|0.244|
|**_Effect in %_**|33.6%|15.2%|34.0%|15.2%|16.0%|4.5%|16.4%|4.5%|15.2%|3.7%|15.6%|3.9%|

_Note:_ Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle, sector, fiscal year when project became effective and country.

Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 34
# Table 8. EFFECT OF IE ON GAP

|||DID|-OLS|||DID|-RE|||DID|-FE||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|**T (quarters with IE==1)**|-0.073***|-0.061***|-0.074***|-0.061***|-0.065**|-0.038|-0.066***|-0.037|-0.066**|-0.034|-0.067**|-0.034|
||(-8.17)|(-5.72)|(-8.28)|(-5.73)|(-2.56)|(-1.48)|(-2.60)|(-1.45)|(-2.43)|(-1.29)|(-2.47)|(-1.27)|
|**D (project with IE==1)**|-0.0098|0.0056|-0.0094|0.0055|-0.0090|-0.0080|-0.0079|-0.0083|||||
||(-1.43)|(0.46)|(-1.37)|(0.45)|(-0.51)|(-0.33)|(-0.44)|(-0.34)|||||
|**_TTL switch dummies_**|Y|N|Y|N|Y|N|Y|N|Y|N|Y|N|
|**_TTL UPI dummies_**|N|Y|N|Y|N|Y|N|Y|N|Y|N|Y|
|**_Polynomial of sector averages_**|N|N|Y|Y|N|N|Y|Y|N|N|Y|Y|
|**_Observations_**|17784|17784|17784|17784|17784|17784|17784|17784|17784|17784|17784|17784|
|**_Outcome mean_**|0.148|0.148|0.148|0.148|0.148|0.148|0.148|0.148|0.148|0.148|0.148|0.148|
|**_Effect in %_**|49.3%|41.2%|50.0%|41.2%|43.9%|25.7%|44.6%|25.0%|44.6%|23.0%|45.3%|23.0%|

_Note:_ Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle, sector, fiscal year when project became effective and country.

Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 35
Table 9. EFFECT OF IE ON RATE OF DISBURSEMENT ADJUSTED BY SECTOR AVERAGE

|||DID|-OLS|||DID|-RE|||DID|-FE||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|**T (quarters with IE==1)**|0.74***|0.48***|0.74***|0.48***|0.46***|0.40***|0.44***|0.40***|0.42***|0.41***|0.40***|0.40***|
||(5.60)|(4.03)|(5.54)|(4.00)|(3.88)|(3.03)|(3.68)|(2.93)|(3.62)|(3.04)|(3.39)|(2.91)|
|**D (project with IE==1)**|-0.46***|-0.21|-0.46***|-0.21|-0.30|-0.21|-0.29|-0.21|||||
||(-3.84)|(-1.53)|(-3.81)|(-1.53)|(-1.44)|(-1.02)|(-1.38)|(-0.99)|||||
|**_TTL switch dummies_**|Y|N|Y|N|Y|N|Y|N|Y|N|Y|N|
|**_TTL UPI dummies_**|N|Y|N|Y|N|Y|N|Y|N|Y|N|Y|
|**_Polynomial of sector averages_**|N|N|Y|Y|N|N|Y|Y|N|N|Y|Y|
|**_Observations_**|17784|17784|17784|17784|17784|17784|17784|17784|17784|17784|17784|17784|
|**_Outcome mean_**|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|1.00|
|**_Effect in %_**|74.0%|48.0%|74.0%|48.0%|46.0%|40.0%|44.0%|40.0%|42.0%|41.0%|40.0%|40.0%|

_Note:_ Standard errors are clustered (sector and quarter level). T-stats are in parenthesis. All specifications include fixed effects for quarters of project life cycle, sector, fiscal year when project became effective and country.

Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent 36
<!-- Start of picture text --> 80<br>© Water<br>50 y ey| . msP<br>20 i<br>10 aNN m Energy<br>—<br>0 m Edu<br>1 3 5 7 9 114 13 15 17 19 21 23 25 27<br><!-- End of picture text -->

<!-- Start of picture text --> 100%<br>_<br>90% ps ee_ —_ ~~ jj © Water<br>80% — ~~. ~ N i © Urban<br>70% Vw © Transport<br>60% mTMSP<br>50% SDV<br>40% § Health<br>30% mFPD<br>20% HEnv<br>10% @ Energy<br>0% m Edu<br>13 5 7 9 141 13 15 17 19 21 23 25 27<br><!-- End of picture text -->
<!-- Start of picture text --> 40<br>— | Water<br>YN | Transport<br>Wy —<br>AY mSDV<br>Ne = Health<br>1s \- mFPD<br>5 SY. Energy<br>mEdu<br>0 m Agr<br>12345 6 7 8 9 101112131415 1617 18 19 2021<br><!-- End of picture text -->

<!-- Start of picture text --> 90% wee a Water<br>70% © Transport<br>msSP<br>60%<br>m= SDV<br>50%<br>@ Health<br>40% mFPD<br>30% mEnv<br>20% M Energy<br>10% mEdu<br>0% mAgr<br>123 45 6/7 8 9 101112131415161718192021<br><!-- End of picture text -->