# **Notes**

Research Manager, Development Research Group, World Bank; and William R. Kenan, Jr. Professor of Economics, Department of Economics, University of Pennsylvania.

1 For instance, the International Initiative for Impact Evaluation (3IE) is being set up by governments of several countries, donor agencies and private foundations to address the desire of the development community to build up systematically more evidence about effective interventions.

2 There have been excellent reviews of the choice of methods as applied to social programs. See, for example, Grossman (1994), Heckman and Smith (1995), Ravallion (2001), Cobb-Clark and Crossley (2003), and Duflo (2004).

3 To keep the discussion focused on the timing issue and the duration of exposure, we avoid discussing the specific evaluation method (or methods) that is used by the empirical studies that we cite. However, we restrict our selection of studies to review to those that have a sound evaluation design, whether experimental or using econometric techniques. We also do not discuss estimation issues such as sample attrition bias which is one of the ways in which a duration issue has been taken into account in the evaluation literature.

4 See Heckman, Lalonde and Smith (1999) for a review.

5 Because Rouse and Krueger (2004) define the treatment group more stringently, however, the counterfactual treatment received by the control students becomes more mixed, and a share of these students is contaminated by partial participation in the program.

6 In their assessment of the returns to World Bank investment projects, Pohl and Mihaljek (1992) cite construction delays among the risks that account for a wedge between ex ante (appraisal) estimates and ex post estimates of rates of returns. They estimate that, on average, projects take considerably more time to implement than expected at appraisal: six years rather than four years. 7 In Mexico’s well-known PROGRESA program, payment records from an evaluation sample showed that 27 percent of the eligible population had not received benefits after almost two years of program operation, possibly a result of delays in setting up the program’s management information system (Rawlings and Rubio 2005). In Ecuador’s _Bono de Desarrollo Humano_ , the lists of the beneficiaries who had been allocated the transfer through a lottery did not reach program operators and so about 30 percent of them did not take up the program (Schady and Araujo 2008).

8 Chin (2005) makes a one-year adjustment in her evaluation of Operation Blackboard in India. Although the Indian government allocated and disbursed funds for the program for the first time in fiscal year 1987, not all schools received program resources until the following school year. In addition to the delay in implementation, Chin also finds that only one-quarter and one-half of the project teachers were sent to one-teacher schools while the remaining project teachers were used in ways the central government had not intended. Apparently, the state and local governments had exercised their discretion in the use of the OB teachers.

9 In two programs that we know, administrative records at the individual level were maintained at local program offices, not at a central program office, and local recordkeeping varied in quality and form (for example, some records were computerized and some were not), so that a major effort was required to collect and check records during the evaluations.

31