metadata: authors: 0: author: Pereira Guimaraes Leite,Phillippe George
1: author: Narayan,Ambar
2: author: Skoufias,Emmanuel
count: World,Mexico
docna: 0: docna: How do ex ante simulations compare with ex
            post evaluations ? evidence from the impact of conditional
            cash transfer programs
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Labor Markets,Disability,Services & Transfers to Poor,Gender and Development,Economic Assistance,Access of Poor to Social Services,Inequality,Rural Labor Markets
teratopic: Governance,Poverty Reduction,Science and Technology Development,Education
abstracts: cdata!: This paper compares the ex ante
            simulation of the impacts of conditional cash transfer
            programs against the ex post estimates of impacts obtained
            from experimental evaluations. Using data on
            program-eligible households in treatment areas from the same
            baseline surveys that are used for experimental evaluations
            of conditional cash transfer programs in Mexico and Ecuador,
            the authors use a micro-simulation model to derive ex ante
            estimates of the impact of the programs on enrollment rates
            and poverty. The estimates reveal that ex ante predictions
            of certain impacts of conditional cash transfer programs
            match up well against the benchmark estimates of ex post
            experimental studies. The findings seem to support the use
            of this model to assess the potential impact and cost
            efficiency of a conditional cash transfer program ex ante,
            in order to inform decisions about how the program would be designed.
geo_reg_key: 119228,517191
repnme: repnme: How do ex ante simulations compare with ex
            post evaluations ? evidence from the impact of conditional
            cash transfer programs
display_title: How do ex ante simulations compare
            with ex post evaluations evidence from the impact of
            conditional cash transfer programs
listing_relative_url: /research/2011/06/14447220/ex-ante-simulations-compare-ex-post-evaluations-evidence-impact-conditional-cash-transfer-programs
topicv3: Labor Markets m1326488 855,Poverty m1326687 695,Cash Transfers m1327437 1608,Gender m1327621 615,Health m1504661 3480
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Governance,Poverty Reduction,Science and Technology Development,Education
no_of_pages: 50
author_keys: 0: author_key: 605170
1: author_key: 167324
2: author_key: 177660
model_extractions: WPS5705 Policy Research Working Paper 5705 How Do _Ex Ante_ Simulations Compare with _Ex Post_ Evaluations? Evidence from the Impact of Conditional Cash Transfer Programs _Phillippe Leite Ambar Narayan Emmanuel Skoufias_ The World Bank Poverty Reduction and Economic Management Network Poverty Reduction and Equity Unit June 2011
Policy Research Working Paper 5705

# **Abstract**

This paper compares the _ex ante_ simulation of the impacts of conditional cash transfer programs against the _ex post_ estimates of impacts obtained from experimental evaluations. Using data on program-eligible households in treatment areas from the same baseline surveys that are used for experimental evaluations of conditional cash transfer programs in Mexico and Ecuador, the authors use a micro-simulation model to derive _ex ante_ estimates of the impact of the programs on enrollment rates and poverty. The estimates reveal that _ex ante_ predictions of certain impacts of conditional cash transfer programs match up well against the benchmark estimates of _ex post_ experimental studies. The findings seem to support the use of this model to assess the potential impact and cost efficiency of a conditional cash transfer program _ex ante,_ in order to inform decisions about how the program would be designed.

This paper is a product of the Poverty Reduction and Equity Unit, Poverty Reduction and Economic Management Network. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http:// econ.worldbank.org. The author may be contacted at eskoufias@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
How Do _Ex Ante_ Simulations Compare with _Ex Post_ Evaluations? Evidence from the Impact of Conditional Cash Transfer Programs

# Phillippe Leite

# Ambar Narayan

Emmanuel Skoufias<sup>1</sup>

**Keywords:** Experiments, Conditional Cash Transfers, Mexico, Microsimulation, Ecuador. **JEL Classification:** C21, C52, I38, I2, I1

1 Phillippe Leite is an Economist in the Human Development Network (HDNSP) of the World Bank. Ambar Narayan and Emmanuel Skoufias are a Senior Economist and a Lead Economist, respectively, in the Poverty Reduction and Equity unit (PRMPR) in the PREM network of the World Bank. The authors are grateful to Francisco Ferreira for his input in the early stages of this work, to Tarhat Shahid for excellent research assistance, and to the Multi-Donor Trust Fund for Poverty & Social Impact Analysis for partial support. The authors share equal responsibility for the contents of this paper. The findings, interpretations, and conclusions are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represent.
# **I. Introduction**

Conditional Cash Transfer programs (CCTs) have been used by many governments in their attempts to reduce poverty and inequality and achieve longer term human development. CCTs are typically intended to generate incentives among the poor to improve human capital so that they can break free of poverty traps across generations. While CCT programs in different countries may vary in their design and purpose, they tend to share a few basic features. All of them involve transferring resources (usually in the form of cash) to poor households, which are usually provided directly to the mother with reciprocal conditions imposed on the household, to encourage changes in behavioral norms. These conditions commonly include requiring, for example, that children attend school and/or individuals have regular health facility visits.

Different methods of evaluation have been developed to analyze the likely impacts of a monetary transfer on human capital accumulation and poverty alleviation. The array of literature on program evaluation presents two broad types of analysis, _ex post_ and _ex ante_ . _Ex post_ evaluations can in turn be divided into two categories: experimental and quasiexperimental. Experimental analyses are those where households or geographic units are randomly assigned to both treatment and control groups prior to the implementation of the program. Quasi-experimental evaluations are those where treatment and control groups are _not_ selected through random assignment (of households or geographic units), with the result that statistical techniques are necessary to correct and control for observable and unobservable differences between the two groups that might affect outcomes of interest.

_Ex post_ evaluation methods have major advantages in terms of generating statistically valid results about the impact of a monetary transfer through a CCT program, against a counterfactual of not having the program. However, such methods can also have a few important disadvantages, depending on the type of techniques used. Experimental _ex post_ evaluations may provide more statistically valid and robust results relative to quasiexperimental methods, but they are often costly and time-consuming to implement, particularly on a large scale. Quasi-experimental techniques, while typically less costly and easier to implement than experimental analyses, have the disadvantage of requiring a high level of statistical complexity to account for selection bias and the effects of unobservable characteristics.

2
Regardless of the method used, an exclusive reliance on _ex post_ evaluations is also likely to provide little evidence prior to implementation about the _potential_ impact of programs– a question that policymakers are often most interested in, while designing large programs or reforms. Often, the design and implementation of large income transfer programs cannot be delayed till a series of pilots “testing” various design elements have been completed and evaluated using _ex post_ techniques. In other cases, even when the pilot(s) have been tested and evaluated, concerns about “external validity” imply that the policymaker cannot be sure whether a program implemented on a much larger scale than the pilot would have similar impacts or lack thereof. Moreover, policymakers are often interested in questions like the sensitivity of outcomes to amounts of transfers, the differences between the impacts of conditional and unconditional transfers, or the efficiency and effectiveness of alternative program designs. While testing alternative program designs is possible in theory using _ex post_ methods, the complications that arise in implementing and evaluating the relative impacts of multiple “treatments” make such evaluations extremely difficult in practice.

_Ex ante_ evaluations involve _simulating_ the effects of a program on the basis of a household model, in most cases using a data set representative of all program beneficiaries. Such techniques, while having the disadvantage of not measuring the actual impacts of a program, can be particularly useful in addressing some of the questions mentioned above. At the same time these techniques, which rely on structural models of economic behavior, are often criticized about the strong underlying assumptions that are necessary. Bourguignon, Ferreira and Leite (2003) (henceforth BFL) propose such a behavioral model that allows policymakers to evaluate the potential effect of a CCT on poverty, inequality and school enrollment rates. The model combines arithmetic methods with models of demand for schooling, generating predictions about the potential occupational choice (one of which is the decision of going to school) of a child living in a region where a CCT is implemented. The BFL model generates estimates of likely effects of a monetary transfer program, comparing the simulated occupational choice of children with the status quo (absence of the CCT), keeping everything else constant.

The evaluation literature in most part considers _ex ante_ and _ex post_ methods as competing or substitute methods. Ravallion (2008) underscores that the two approaches can instead be complementary – combining an _ex post_ evaluation with an _ex ante_ structural model of schooling 3
choices would allow policymakers to expand the existing set of policy alternatives in considering the optimal design of a given program. While this is true in principle, in order for the two approaches to complement each other in the context of a specific program, the consistency of the results produced by the two approaches would be an important consideration. The concerns about the validity of assumptions that underlie the _ex ante_ behavioral models make it all the more important that their predictions are checked for consistency with the results of _ex post_ experimental evaluations. Any evidence to suggest that _ex ante_ approaches yield results that are at least broadly consistent with those from methodologically sound experimental evaluations would strengthen the case for using such approaches to complement _ex post_ evaluations of similar programs in future, where the relative ease of employing _ex ante_ approaches can be particularly useful in evaluating alternative design options before a program is implemented.

Our paper represents an attempt to generate evidence on the consistency between the two approaches in the context of CCT programs by comparing _ex ante_ predictions with results of _ex post_ experimental evaluations, following the suggestion of Bourguignon and Ferreira (2003). Specifically, we compare the school enrollment, poverty and inequality impacts of a CCT program generated _ex ante_ , using the behavioral model suggested by BFL, with the corresponding impacts estimated _ex post_ from a randomized experiment involving the same program. The exercise is conducted for a number of CCT programs, of which the results of two exercises are presented in detail.

The comparison presented in this paper is a validation test of the strongest hypothesis of the BFL model: that the cross-sectional income effects estimated _ex ante_ with a representative household sample would coincide with the _ex post_ estimates of income effects generated by the program. The _ex ante_ simulation of the impact of CCTs on enrolment rate and poverty is conducted using the same baseline surveys that are used for the experimental evaluations of _PROGRESA_ in Mexico and _Bono de Desarrollo Humano_ (henceforth BDH) in Ecuador. The full set of _ex ante_ predictions are then compared with the highly credible and accepted _ex post_ results presented in Skoufias and Parker (2001) and Schultz (2000) for _PROGRESA_ and Schady and Araújo (2008) for BDH. The choice of these programs for the exercise is partly to do with the fact that they represent large and well-known CCT programs that contain many of the typical features of such programs. Equally important for our purposes is that for both these programs, the randomization of program implementation allowed researchers to conduct 4
methodologically sound experimental evaluations. As a result, the results of these evaluations provide an appropriate benchmark against which _ex ante_ predictions using the BFL model can be compared. The use of the baseline data from these evaluations to generate the _ex ante_ predictions ensures that the _ex ante_ simulations are based on the very same household sample on which the program impacts are estimated _ex post_ .

Section II below reviews the literature most relevant to our analysis. Section III provides a synopsis of the BFL model, including its underlying assumptions and how it works. Section IV discusses the two CCT programs on which the BFL model will be applied ( _PROGRESA_ and BDH), the data to be used for the analysis and the _ex post_ evaluation studies that provide benchmark results. Section V presents the results of the comparison between the predictions of the BFL model and results of _ex post_ evaluations. Section VI concludes the paper.

# **II. Review of the Literature**

Our review of relevant literature consists of two related parts. Given the topic of our paper, it is important to discuss earlier work done by a number of researchers comparing results from _ex ante_ models and _ex post_ evaluations. Since we use results from _experimental ex post_ evaluations as the yardstick to test the validity of _ex ante_ predictions, it is also important to look back at some of the seminal literature, going back to a few decades, concerned with the predictive power of non-experimental estimators against the benchmark of experimental evaluation results.

The act of forecasting outcomes is frequently subject to criticism, as it relies on estimation of parameters that are sensitive to model specification and to non-observable errors. Forecasts about the impacts of social programs are typically validated by comparing results with the estimated treatment effects from randomized experiments when they are available. Examples of such analyses include Todd and Wolpin (2004, 2006), Attanasio et al (2002) and de Janvry and Sadoulet (2006) for the Mexican case and Lise et al (2003) for the Canadian Self-Sufficiency Project (SSP).<sup>2</sup>

> 2 The SSP is a social program providing time-limited earnings supplements to Income Assistance recipients who are able to obtain full time employment within a 12 month period.

5
Todd and Wolpin (2006) use an _ex ante_ model to generate child schooling estimates that compare reasonably well with experimental results. For children aged 12-15 years, the model predicts the effects of _PROGRESA_ on school enrollments to be in the same direction as experimental results, but underestimates them by 21-25 percent. However, when the sample is disaggregated into age groups of 12-13 years and 14-15 years, the results compare less well, particularly among boys. The _ex ante_ model does not predict any change in outcomes for boys aged 12-13 years and significantly overestimates the effects of the program for the 14-15 years age group. Todd and Wolpin also use a particular sample that is not comparable with the one used by Skoufias and Parker (2001) and Schultz (2000) to generate _ex post_ results. They restrict their sample of analysis from the original 24,077 households surveyed under _PROGRESA_ to only landless households where the spouse of the household head is a woman under 50 years of age, reducing their data sample to 3,401 households (and then by 209 more due to data problems).

Attanasio et al (2005) present a complex model to simulate _ex ante_ the likely effects of _PROGRESA_ , as well as that of a hypothetical program that differs from _PROGRESA_ in the way the transfers vary by the grade attended by the child. They do not compare the results of the first exercise with _ex post_ results and instead only present the _ex ante_ program impacts for different age groups. According to our computations using their estimates, the change in enrollment due to _PROGRESA_ predicted by their model translates to enrollment increases of approximately 2 and 5.4 percent among children of age 10-13 years and 14-17 years respectively.<sup>3</sup> In comparison, the enrollment increases attributed to _PROGRESA_ – as estimated by the _ex post_ evaluations of Skoufias and Parker (2001) and Schultz (2000) – are 2.4 and 7.5 percent for children of age 10-13 years and 14-17 years, respectively. The Attanasio et al model is thus able to generate predictions that match up quite well with _ex post_ evaluation results, even as its complexity and challenges of implementation are considerable.<sup>4</sup> Leite (2007) applies the BFL model to data from PNAD ( _Pesquisa Nacional por Amostra de Domicílios_ ) 1999 in Brazil to forecast an increase of 3.9 percent in the enrollment rate of poor

> 3 Our computations involve extrapolating from the enrollment difference presented in Figure 1 of Attanasio et al (2005), by assigning weights to each age group in Figure 1 and using the baseline survey. 4 These authors also propose the use of a structural model to analyze issues that remain unexplored by standard Difference-in-Difference (henceforth DID) estimators. For example, they estimate that the performance of _PROGRESA_ could have been improved by offering more resources to older age cohorts of children such as secondary school students and less to younger age cohorts or primary school students.

6
children of age 10-15 years. This result is in the ballpark of the actual 3 percent increase in enrolment for the same cohort of boys between 1999 and 2003 estimated from PNAD 2003, which is _not_ a result from an impact evaluation. The forecast from the BFL model is also close to the results from a quasi-experimental evaluation of the Bolsa Escola program by Cardoso and Souza (2004), in which a matching estimator suggests a 3.1 percent increase in the enrollment rate in boys and 3.0 percent increase for girls aged 10-15 years associated with the Bolsa Escola. De Janvry and Sadoulet (2006) also use _PROGRESA_ data to develop a predictive model. They analyze the potential for increase in school attendance in order to help increase the efficiency of such transfers, concentrating on secondary school attendance and the extent to which it can be affected by a conditional transfer.

Lise et al (2003) work with data from the Canadian Self-Sufficiency Project (SSP) and construct a dynamic, partial equilibrium model to simulate the effects of the program in terms of labor market behavior and compare these to observed treatment effects on individuals. The SSP is a social program providing time-limited earnings supplements to Income Assistance recipients who are able to obtain full time employment within a 12 month period. Recipients (i.e. the treatment group) of SSP were picked at random from among those eligible for these benefits. Lise et al restrict their sample to single mothers only, with around 2,300 women in the control and treatment groups alike.<sup>5</sup> Their _ex ante_ model is calibrated to control group data and the experiment is simulated within this model to imitate the “welfare-to-work transition of the treatment group.”<sup>6</sup> The results are then compared with those of an _ex post_ evaluation of the SSP, which produces similar results.

An earlier body of literature, including LaLonde (1986), Heckman and Hotz (1989) and others, use experimental results as benchmarks for evaluating the predictive power of partial equilibrium non-experimental estimators, mainly in the context of labor market programs. These papers concluded that non-experimental methods were not effective in evaluating program impacts. In more recent literature, propensity score matching was extended with

> 5 Lise et al’s calibrated search-matching model incorporates three segments of the market: employed individuals, unemployed individuals receiving unemployment benefits, and individuals receiving income supplements through IA. The model expands on Davidson and Woodbury (1993)’s equilibrium search model by assuming that expected lifetime income is maximized by individuals when they choose their employment state and the intensity with which they seek work if unemployed.

> 6 For further details, please see Lise, Seitz and Smith (2003)

7
kernel and local linear matching estimators, which use multiple nonparticipants to estimate the outcomes of the control group as opposed to pair-wise estimation. This method, used by Heckman et al. (1997) and Heckman et al. (1998), was implemented with longitudinal and crosssectional data from the National Supported Work Demonstration, another labor market program.<sup>7</sup> The results suggest that such estimators are able to replicate experimental results only when the data examined are very similar – from the same source(s), with treatment and control groups from the same geographic labor markets, and incorporating a range of variables that influence both participation and labor market outcomes.<sup>8</sup> McKenzie et al (2006) build on the finding of previous authors: the best non-experimental estimates are obtained when the treatment and control groups are drawn from the same labor markets, data sources used are the same, and groups’ characteristics are similar in that the “likelihood of receiving the treatment is similar in both groups.” They compare experimental and non-experimental methods in analyzing the income gains from migration, using a survey on the quota of Tongans allowed to immigrate to New Zealand each year on the basis of a lottery from an excess number of applicants. They generate experimental estimates of income gain from migration by comparing gains of applicants whose names were not drawn in the lottery with those who immigrated through the lottery, after taking into account those who did not immigrate even after they “won” the lottery. They also survey a non-applicant group and generate non-experimental estimates for comparison with the experimental estimates. Because migrants are more likely to have certain observable _and_ unobservable characteristics (such as ability and drive), the non-experimental methods are shown to overstate income gains from migration by between 9 and 82 percent.

The _ex ante_ labor market models described here suggest the difficulties in predicting the impacts of a program using a model representing standard economic incentives incorporated through wages. In our paper, however, the effects of programs are simulated using a model incorporating _household_ income, where the incentives operate through channels that are quite different.

> 7 NSW was a subsidized work experience program providing recipients with training and assistance in finding regular jobs.

> 8 Summarized from Smith and Todd (2006).

8
Against the backdrop of the existing literature discussed above, the main contribution of our paper is in terms of testing the validity of an _ex ante_ tool for evaluating the impacts of CCT programs (the BFL model) that is operationally useful, both in terms of ease of implementation and data requirements. Compared to the dynamic _ex ante_ models suggested by Attanasio et al and Todd and Wolpin (see above), the BFL model makes fewer demands on data requirements and do not rely on the availability of panel data, as it employs a reduced-form approach that imposes simplifying assumptions on household behavior. If the predictions of the BFL model with all its simplicity were to compare well with the impacts measured by rigorous _ex post_ evaluations, it would be a practical and credible _ex ante_ tool for policymakers to employ during the design and planning phase of CCT programs.

# **III.** **_Ex Ante_ Simulation with the BFL Model**<sup>**9**</sup>

This section presents a summary description of the BFL model, used as the _ex ante_ evaluation tool in this paper (a more detailed discussion of the model can be found in the original BFL paper). Rather than constructing a complete structural model of demand for schooling and intra-household labor allocation, the BFL model aims to obtain reasonable orders of magnitude for the likely effects of cash transfers that are conditional on school attendance. The structural aspects of modeling are thus kept to the minimum necessary to capture the main effects of the program. There are four key underlying assumptions. Firstly, occupational choice is assumed to reflect (in reduced form) the end result of whatever decision-making process about a child’s time allocation unfolds in a household, regardless of _how_ the decision is made. Secondly, the decision to send a child to school is assumed to be made _after_ all occupational decisions by adults within the household have been made, so that the schooling decision does not affect those earlier decisions. Thirdly, the model does not allow for decisions about the occupational choice of multiple siblings in a household to be made simultaneously or jointly. Fourthly, the composition of the household is taken as exogenous. While each of these assumptions is an

> 9 A toolkit for running the BFL model was prepared by the World Bank. The toolkit provides a how-to guide in the application of the BFL micro-simulation model that enables users to analyze and compare several alternative scenarios on the basis on representative household survey data. It can thus be used in selecting the most cost-effective design or it can be used for sensitivity and cost-benefit analysis. The toolkit can be downloaded at www.worldbank.org/safetynets under _toolkit_ or by clicking on the following hyperlink: BFL model toolkit

9
abstraction from reality (at least for some households), they make for a simple, reduced-form approach that is tractable with the baseline data available for most programs.

Let _j_ =0 be the occupational category of “not attending school,” _j_ =1 be that of “attending school and working,” and _j_ =2 be that of “attending school only.” Following the approach adopted in BFL, the utility function of child _i_ corresponding to each occupational category _j_ is given by the following:<sup>10</sup> Where _Zi_ are the characteristics of both the child and the household, _Y-i_ is the household income without the child’s earnings, _yij_ is the child’s income earned in alternative _j_ ; and _vij_ is the random variable representing idiosyncratic preferences. _γj_ and _αj_ are parameters specific to occupational category _j_ . The model is parsimonious in its representation of occupational choice of children and allows for all possible trade-offs between current income of the household and the schooling of the child.<sup>11</sup> A key variable is the child’s earning _yij_ since transfers depend on household income and the schooling/occupational status of a child, which in itself affects the child’s earning.

The model implicitly treats the child's number of hours of work as a discrete choice between 3 alternatives, corresponding to the 3 occupational categories given by _j_ . It seems reasonable to define child _i_ ’s earning in occupational category 0 as equal to her observed market earnings denoted by _wi_ , since no time is spent on schooling. Assuming that earnings can be characterized by the standard Becker-Mincer human capital model, _wi_ is given by: Where _Xi_ is a set of individual characteristics (including age and schooling achieved) and _ui_ is a random term that stands for unobserved determinants of earnings. The second term on the right hand side is a dummy variable representing the fact that a child who attends school and

> 10 See equation (2) in the BFL paper, which in turn is the linearized version of a more general specification given by equation (1) in their paper.

> 11 Note that the model can also implicitly represent a trade-off between current income of the household and future income of the child (and perhaps the household as well), since schooling would be expected to raise his/her future income

10
works for wages has less time available and is therefore likely to earn less. The child's contribution to the household income ( _yij,_ ) in the various alternative categories for _j_ is given by: Where _yij_ covers income from both market-based and domestic work done by child _i_ in occupational category _j_ . Being in category 1, namely attending school while working outside the household, leads to a reduction in total income (relative to income in category 0) by the proportion (1-M). Similarly, going to school without working in the market leads to a reduction in total income by the proportion (1-D).<sup>12</sup> While D is not observed, M is assumed to be the same for domestic and market work and can be estimated from observed earnings represented by equation (2) above. Replacing (3) in equation (1): Child _i_ (or the household of child _i_ ) will choose the occupational category j that yields the highest utility among the 3 alternatives. Assuming that all parameters of equation (2) are known, along with actual or potential earnings ( _wi_ ) and the residual terms ( _vij_ ), household’s selection of a child’s occupation can then be defined by: If a CCT program is implemented where _all_ children going to school receive a transfer T, equation (4) becomes: Equation (6) adds a transfer amount T to the part of household income that is independent of the child’s income, conditional on the child going to school. Thus in maximizing the utility function given by (6), the household must take into account the fact that it will receive T only in states (j=1) or (j=2). Under the assumptions made earlier, equation (6) is the full reduced-form model of the occupational choice of children, which would allow for simulations of the impact of

> 12 Note that total income in category 1 can be a combination of incomes from market and domestic work of the child, while total income in category 2 is from domestic child work only.

11
CCT transfers on those choices, once the estimates of the parameters of the model as well those of _wi_ and _vij'_ s are obtained.

# _Estimation of discrete choice model_

An assumption that the _vij_ are independently and identically distributed across sample observations with a double exponential distribution leads to the well-known multinomial logit model, which can be used to estimate equation (6), where j=0, 1, 2 are the three alternative categories of occupation the household can choose between. The estimation of parameters is complicated by the nature of the model, which only allows the coefficients corresponding to a given category to be estimated as a _deviation_ from those of a reference category. To see this, we start by noting that the probability a child/household _i_ will select occupational choice _k_ is given by: Taking (j = 0) as the reference state or occupation category, (7) can be written as: Multinomial logit estimation permits the estimation of only the differences _(αj-α0)_ , _(βj-β0)_ , and _(γj-γ0)_ for j = 1, 2. Since the transfer is conditional on the child being in state 1 or 2, the income variable is asymmetric across alternatives. Thus in order to find the utility maximizing alternative (k<sup>*</sup> ), it is necessary to find estimates of the coefficients _α0_ , _α1_ and _α2_ , for which the structural assumptions listed in equation (4) are useful.

Let and be the estimated coefficients of the multinomial logit model corresponding to the income and the child earning variables for alternatives (j = 1) and (j= 2), the alternative (j=0) being taken as the reference. Using equation (4) we get the following system of equations:<sup>13</sup>

> 13 To obtain (9), recall from (4) that: and that: for j=1, 2.

12
Which is equivalent to Using (9a), we can derive estimates of _α0_ , _α1_ , _α2_ and _D_ from the estimated coefficients of the multinomial logit and M; where M is obtained from the estimation of equation (2). In estimating the residual terms ( _vij-vi0_ ), it is important to recall that these cannot be observed in a discrete choice model, and can instead only be known to belong to certain intervals. Thus the residual term for each child is drawn from the relevant interval, which is to say, in a way consistent with that child’s observed choice of occupational category.<sup>14</sup> Finally, we note that equation (6) is not easily estimated without variable _wi,_ which is unobservable for children who are not in the labor market. The most rigorous approach would be to estimate the discrete choice model and the earning equation _simultaneously_ by maximum likelihood techniques – a cumbersome procedure.<sup>15</sup> Correcting the estimation of the earning function for a selection bias turns out to be problematic as well (see discussion in BFL).<sup>16</sup> Instead, the BFL model uses a simple approach, which has the advantage of transparency and robustness. This consists of estimating equation (2) by OLS, which is then used to predict the potential wage _wi_ for children who are not in the labor market. A random term _ui_ is added to the predicted earning of each non-working child to account for unobserved heterogeneity, by drawing from the distribution generated by the residuals of the OLS.

# _Simulating the impacts of a CCT program_

Using the estimation steps described above, the outcomes of a given CCT program can be simulated with equations (5) and (6), but with the additional step of introducing a “means test” to identify eligible beneficiaries, which is a feature in CCT programs like _PROGRESA_ and BDH.

> 14 For instance if child _i_ has chosen state or category 1, the terms (vij-vi0) must be drawn so as to satisfy the inequality: . 15 To handle simultaneously the random terms of the discrete choice model and that of the earning equation, a multinomial probit would then be preferable to a multinomial logit. This would then however pose the difficult challenge of integrating tri-variate normal distributions.

> 16 Proper identification using the inverse Mill’s ratio to correct the earnings equation for election bias requires variables/instruments which influence earnings but not the occupational/schooling choice of children.) Such variables are not readily available from the data. Moreover, the standard correction using a two stage procedure is even weaker in the case of more than two choices.

13
The means test is represented by assuming that the transfer T is provided only if household income is less than or equal to a pre-determined threshold _Y0_ . Taking into account both the means-test and the conditionality of child attending school, child (household) _i_ would choose the state or occupational category that yields the maximum utility among the following alternatives: As explained earlier, the estimation of the multinomial logit regression, combined with the relationships shown in (9a), allows us to simulate the utility maximizing decision for household _i_ to choose among the alternatives specified in (10). It is easy to see that the introduction of a transfer can induce households to move from occupational category 0 (no schooling) to category 1 or 2, and also from 1 to 2 if it were the case that the household qualifies for the transfer only when the child went to school _and_ stopped working.

The framework in (10) can be used to simulate the impacts of a variety of CCT programs that are conditional upon schooling enrollment, allowing for both the means test and transfer to be dependent on individual or household characteristics (e.g. transfer amount varying according to age or gender of the child). That said, important caveats or limitations apply to this framework, which are closely related to the assumptions set out at the beginning of this section. Firstly, the model cannot account for the effects of any upper limit on transfers going to a single household, which is a direct result of the model ignoring the possibility that decisions affecting multiple children in a household may be made simultaneously. Secondly, household income excluding the earnings of the child is treated as exogenous. This does not take into account for the possibility of the means test affecting adult labor supply – for example, when an adult decides to not participate in the labor market if the extra income makes the household ineligible for the CCT program. While this can be a serious issue when eligibility is defined by actual (or observed) 14
income, it is less so when eligibility is defined by a score-based “proxy” means test that attempts to reflect permanent income.<sup>17</sup>

# **IV. Programs and Data Description**

_PROGRESA_ was launched in Mexico at a time when economic growth was found to be insufficiently pro-poor and existing safety net programs were seen as ineffective (Coady, 2004). _PROGRESA_ was designed to alleviate poverty in both the short and long run, by making monetary transfers to poor families while requiring that they invest in the human capital of their children. Renamed Opportunidades in 2003, _PROGRESA_ provides monetary grants to selected families (and usually to the mothers) conditional on children being enrolled in and regularly attending school, and on all family members attending scheduled visits to health care centers.<sup>18</sup> The educational grant is provided for each poor child under the age of 18 years and enrolled in school between the 3<sup>rd</sup> grade of primary and 3<sup>rd</sup> grade of secondary level. The grant amount is adjusted every six months for inflation and varies by school grade and gender, increasing as children progress to higher grades (see Appendix, Table A-1). The grant is intended to reverse two observed tendencies among poor Mexican communities: older children are more likely to work, which implies that the opportunity cost of going to school increases with the grade level, and girls have higher dropout rates than boys at the secondary level. The health and nutrition grants components are intended to improve health indicators through regular visits to health care centers for all family members and enhance food consumption through nutritional supplements, especially for children under the age of 2 years and pregnant and breastfeeding women.<sup>19</sup>

> 17 Typically the proxy means test score is determined by factors such as ownership of durable goods and assets like land, housing conditions, education and occupational status of household members, and demographic characteristics like number of children and dependency ratio – none of which are directly influenced by the decision of an adult member to participate in the labor market.

> 18 For a more detailed description of _PROGRESA_ see Skoufias and Parker (2001) or Parker and Skoufias (2000).

> 19 Skoufias and Parker (2001) present this component as two components: (i) health, providing basic healthcare for all members of the family without including any monetary transfer, and (ii) nutrition, including a fixed monetary transfer.

15
_PROGRESA_ was first implemented during the first half of 1998 in 320 randomly assigned rural villages out of 506 villages pre-qualified for immediate participation.<sup>20</sup> The remaining 186 villages were assigned as control villages and received their first monetary transfers in December 2000. Implementation of the program in a phased manner and random selection of program villages for the first phase of implementation were extremely important factors aiding rigorous evaluations of the program.<sup>21</sup> The baseline survey ( _Encuesta Évaluation de los Hogares_ or ENCEL) for _PROGRESA_ was collected in 1997 in all 506 pre-qualified villages, covering 24,077 households in seven states.<sup>22</sup> _PROGRESA_ administrators ran a follow-up survey over the same set of households once every six months, generating a rich household panel dataset. Many evaluations of _PROGRESA_ were conducted on the basis of these surveys, including the previously mentioned studies by Skoufias and Parker (2001), Parker and Skoufias (2000), Schultz (2004), Todd and Wolpin (2006), Attanasio et al. (2005), and Coady (2004). Overall results show an increase in the enrollment rates of both boys and girls at the primary and secondary levels, a reduction in the incidence of illness among children of age 0-5 years, an increase in the annual growth rate of children of age 12-36 months and a reduction in all poverty indices. _PROGRESA_ (Opportunidades) is now covering around 5 million households (18 percent of the country’s total population) with a budget equivalent to 0.4 percent of GDP, contributing around 20 percent of the income of participating families.

In 2003, Ecuador updated _Bono Solidario_ , its existing social program focusing on poverty reduction that had been created in 1999. The new program, _Bono de Desarrollo Humano_ (BDH), is part of the Social Protection Plan of the _Ministerio de Bienestar Social_ and re-targeted the transfers under the erstwhile _Bono Solidario_ program. While the main purpose of BDH is to transfer cash to poor households, the transfers are also intended to be conditional on child enrollment and health care center visits. In its early days, compliance with the conditions was not monitored and consequently, non-complying households were not penalized. In order to

> 20 These villages were randomly selected out of 6,396 villages (4,546 considered to be in the treatment group and 1,850 in the control group) using probability proportional to size. Villages were identified on the basis of a community score based on information available from national census data regarding characteristics such as educational levels, occupational composition, and housing conditions.

> 21 Further details are available in Skoufias and Parker (2001).

> 22 The 7 states were: Guerrero, Hidalgo, Michoacan, Puebla, Queretaro, San Luis Potosi, and Veracruz.

16
correct this, program administrators launched a large-scale campaign to stress the importance of compliance.

Under BDH, beneficiary households (again the mothers) receive grants of US$15 per month on the condition that children of ages 6-16 years are regularly enrolled, with at least a school attendance rate of at least 80 percent per month. Poor households with children of ages 0-5 years, additionally, have to fulfill the condition that they make scheduled visits to health centers for growth and development checkups and immunizations.<sup>23</sup> According to Schady and Araújo (2006), the monthly US$15 transfers account for only 7 percent of household expenditure. BDH coverage reached approximately one million households (5 million people), covering 40 percent of the Ecuadorian population with an estimated cost of 0.6 percent of GDP in 2005.

The baseline survey, collected between June and August 2003 for the evaluation of the BDH program, was drawn from four of 22 Ecuadorian provinces around the country.<sup>24</sup> Within all four provinces, “paroquias” (parishes) were randomly assigned. For each selected paroquia, a random household sample was selected, generating a total sample size of 1,488 households. Around 90.5 percent of the household sample had at least one child aged 6 to 17 years at the time of the follow-up survey. None of the households in the sample were enrolled in BDH or _Bono Solidario_ prior to the sample selection. Thereafter, some of the sample households were selected for immediate participation in the program while others were not eligible to receive any transfer in the first two years. The follow-up survey took place between January and March 2005 and collected information on households after program implementation, generating a panel sample of 1,306 households (2,875 children between six and seventeen years of age).

According to Schady and Araújo (2008), around 27 percent of households in both treatment and control groups had stated that school attendance was a prerequisite for receiving BDH grants, despite the fact that the program administrator had never enforced any conditions. BDH was found to have a positive impact on school enrollment and a negative impact on child work; the

> 23 The transfers can be collected at any office of the Banred (the largest network of private banks in Ecuador) or from the National Agricultural Bank. 24 The 4 provinces were Carchi, Imbabura, Cotopaxi, and Tungurahua.

17
fact that households _believed_ school attendance to be mandatory may explain the scale of these program effects.<sup>25</sup>

# **V. Comparison between** **_Ex Ante_ and** **_Ex Post_ Estimates of Program Impact**<sup>**26**</sup>

The BFL model as described in Section III can be used to simulate school enrolment (and labor participation of children) in the presence of the CCT program, which can then be compared with the outcomes for the counterfactual – namely, the observed scenario in the absence of CCT. When the simulation is conducted with the treatment group – a sub-sample of the full baseline survey sample – the impact obtained may be considered as an estimate of the “Average Intent to Treat” (henceforth AIT) because it simulates the impact of the CCT program on the enrollment rate among all eligible children, _regardless_ of whether they accept the transfer or not. Even if a child is eligible to receive the transfer, if the amount of transfer is insufficient to move him/her from category 0 to 1 or 2, the household would not accept the transfer and consequently, the transfer is not allocated to the household in the simulation.<sup>27</sup> The AIT estimator is then given by: Where _P_<sup>_*_</sup> is a dummy variable taking the value 1 if children would be enrolled in school after receiving transfer _T_ and _P_ is a dummy variable taking the value 1 if children are currently enrolled in school. AIT* provides an estimate of the average impact of the availability of the program to eligible households in treatment communities.

Two aspects of the _ex ante_ AIT estimator must be noted. Firstly, it assumes good implementation of program in treatment communities. While the simulation accounts for eligible households opting to not receive the transfer and send their children to school, it does not account for “exclusion errors” in targeting whereby households eligible and willing to

> 25 Interestingly, these results suggest that even if the conditionalities of a CCT program are not enforced, just announcing the conditions may be enough to have some impact on behavior of participants.

> 26 Similar analysis for Nicaraguan Red de Protección Social (RPS) program was performed by the authors and main findings presented in this section also hold for the RPS program. Authors omitted the RPS analysis due to the length of the article but results are available per request (See Appendix, Table A-12 for summary results).

> 27 In maximizing its utility, an eligible household will decide to _not_ accept the transfer if _U(0)_ > _U(1)_ even when _Y-I + M.wi  ≤ Y0_ , and _U(0)_ > _U(2)_ even when _Y-I  ≤ Y0_

18
receive the transfer (complying with its condition of sending children to school) are left out of the program. Neither does it take into account other types of implementation flaws, including “inclusion errors” of non-eligible households receiving the transfers, lack of compliance with the conditions of the transfer, or schools being unable to accommodate increase in enrollments. Secondly, due to the nature of the BFL model, the _ex ante_ AIT estimator does not take into account any time trend effects on outcomes on which impacts are measured. Given this, it is best to compare the _ex ante_ AIT estimate with _ex post_ AIT estimates obtained using doubledifference or difference in difference (DID) method whenever possible – recognizing that this _ex post_ method removes the time trend effect from the estimated impact.

# _The impact of PROGRESA_

We compare the simulated _ex ante_ impact of _PROGRESA_ , applying the BFL model on the baseline survey (from 1997 – see section IV for details), with _ex post_ results from Skoufias and Parker (2001). To make this comparison possible, all income variables from the baseline survey are converted into November 1999 pesos, and transfer amounts corresponding to the second half of 1999 (see Appendix, Table A-1 for the amounts) are applied in the simulations.

The selected sample comprises only children who were of age 8-17 years at the time of the baseline survey in 1997. Given how the BFL model is set up, the data from the follow-up survey on the same children, which is to say the panel data, is not utilized for the simulation. Out of the 33,609 children in the baseline sample the simulation was applied to, around 30 percent were not attending school, while only around 4 percent of children were combining work outside the household with school. Children not attending school were on the average older and with higher education level than those who were in school and not working. Households with higher dropout rates were not necessarily poorer, indicating the importance of child income for the household. Child earnings increased with age and girls were more likely to drop out than boys, but boys were more likely to work and attend school simultaneously (see Appendix, Table A-2 for all descriptive statistics). Average household size was similar (around 7 members) for the 3 groups of children and more educated parents were more likely to have children in school.

Enrollment patterns across age groups are important to note as well, with implications for how program impacts should be estimated. Dropout rates increase sharply around the age when children are expected to complete primary schooling, rising from 7 percent at age 11 to nearly 19
17 percent at age 12 and increasing exponentially thereafter to reach nearly 82 percent at age of 17. Because of the vast differences in enrollment across age groups, Skoufias and Parker suggest that the sample of children aged 8 to 17 years must be divided into two groups in assessing the potential impact of the program: children of primary school age (8-11 years) and those of secondary school age (12-17 years).

The experimental design of _PROGRESA_ ensures in theory that all possible sources of bias were evenly distributed among participants and non-participants, allowing us to strictly attribute differences between treatment and control groups to program effect.<sup>28</sup> However, different papers have analyzed the pre-program composition and characteristics of treatment and control groups in detail, suspecting that despite randomization the two groups were not fully comparable. Skoufias and Parker also find some evidence of systematic differences between the two groups and propose that a double-difference (DID) estimator, which takes into account any pre-existing differences, is preferable in evaluating the impacts of _PROGRESA_ .<sup>29</sup> To be consistent with the Skoufias and Parker’s methodology for _ex post_ evaluation, the BFL model is estimated (see appendix for model specification and estimations) for all boys and girls aged 8-17 years in the baseline sample, and the results are presented separately for three age groups and gender categories. This generates results for six age-gender categories.

Following the argument at the beginning of this section, the _ex ante_ AIT estimator generated by the BFL model (see (11) above) can be compared directly with the _ex post_ results from Skoufias and Parker (Tables 5 and 6), who use the sample of all eligible households and measure the direct effect of the “intent to treat”, regardless of whether they in fact received a transfer or not. The _ex post_ AIT estimates should be seen as a _lower_ bound of the impact on households that actually received treatment, since the observed impact also includes the effect of flaws in program targeting, which would dilute the impact on eligible households (for example, when mis-targeting results in an eligible household being left out of the program). These estimates are derived from a regression-based approach that yields the DID estimate of the program’s impact

> 28 Heckman, La Londe and Smith (1999).

> 29 Also, Behrman and Todd (1999) found the null hypothesis of mean equality with respect to household characteristics between treatment and control group to be rejected more frequently than expected.

20
in each round (of the follow-up surveys), which is net of any preprogram differences between treatment and control households and any time trends in the values of the outcome indicator.

Skoufias and Parker (in Figures 4a and 4b of their paper) show that the mean school attendance rate of both boys and girls were nearly identical for control and treatment villages at the outset of the program. For treatment villages, the enrollment rate among boys was higher than that of girls in the baseline year, where the difference is entirely a result of the higher dropout rate among girls of age 12-17 years (Table 1). For the age group 8-11 years, enrollment was high at around 94 percent for both boys and girls.

**Table 1: Impact of Transfers on Children’s Occupational Choice (1997-1999)** Actual and counterfactual school enrollment rates for _PROGRESA_ target population

|||_Baseline_<br>_(pre-program)_|_Ex post A_|_IT_<sup>_1_</sup>|_Ex ante A_<br>_Conditio_|_IT_<sup>_2_</sup><br>_nal_|_Ex ante AIT_<sup>_2_</sup><br>_Unconditional_|
|---|---|---|---|---|---|---|---|
||8-17 years-old|74.5%|-||4.0%|******|-0.1%|
||||||(0.4%)||(0.7%)|
|Bo|8-11 years-old|93.8%|1.8%|******|0.0%||-0.1%|
|ys|||(0.7%)||(0.4%)||(0.5%)|
||12-17 years-old|57.5%|5.8%|******|5.9%|******|0.0%|
||||(2.1%)||(0.8%)||(0.8%)|
||8-17 years-old|69.4%|-||4.3%|******|-0.1%|
||||||(0.7%)||(0.4%)|
|Gir|8-11 years-old|93.9%|-0.3%||-0.2%||-0.1%|
|ls|||(1.0%)||(0.4%)||(0.4%)|
||12-17 years-old|47.9%|9.5%|******|6.6%|******|-0.1%|
||||(2.2%)||(0.8%)||(0.8%)|

_Source_ : Baseline Survey 1997 and Rounds 1-4;authors' calculation. _Note_ : 1: Results from Skoufias and Parker (2001) - Table 6 (only the results for November 1999 are reported). The coefficients reported are the marginal effects of the _PROGRESA_ program on the probability of attending school; 2: Results from simulation on baseline data using BFL model

** Significant at 5% level; * Significant at 10% level.

Standard errors and standard deviations in parentheses. For _ex post_ AIT, robust standard errors are shown, which account for clustering of individuals within villages. For _ex ante_ AIT, standard deviations computed by bootstrap method.

Table 1 shows the AIT estimates of impact of _PROGRESA_ on school enrollment rates by age and gender (Skoufias and Parker, henceforth SP) as well as the _ex ante_ AIT estimates using the BFL simulation model. _Ex ante_ estimates are also shown for a hypothetical case where the transfer is 21
unconditional, namely provided to all potential beneficiaries, regardless of whether they are enrolled in school or not. The _ex ante_ AIT (conditional) estimates compare quite well with the _ex post_ estimates. Both types of estimates show the impact of _PROGRESA_ on enrollment to be much higher among children of age 12-17 years than children of age 8-11 years, and higher among girls than boys in the 12-17 age group. While the _ex ante_ and _ex post_ estimates are almost identical for boys of age 12-17 and girls of age 8-11, the _ex ante_ estimates understate the impact for girls of age 12-17 and boys of age 8-11. Boys of age 8-11 years are the only group for which the two types of estimates disagree on whether there is any significant impact or not – the _ex post_ estimate show a small positive impact while the _ex ante_ estimate is not different from zero. The confidence intervals of _ex ante_ and _ex post_ estimates for the same group overlay each other to a large extent. One advantage of _ex ante_ methods is that they allow for the simulation of hypothetical scenarios. In the hypothetical case where transfers are unconditional, the simulated effect on enrollment rate is found to be zero for all age/gender groups.

**Table 2: Impact of Transfers on Children’s Occupational Choice (1997-1999)** Actual and counterfactual proportions of children working for _PROGRESA_ target population

|||_Baseline_<br>_(pre-program)_|_Ex post AIT_<sup>_1_</sup>|_Ex ante A_<br>_Conditio_|_IT_<sup>_2_</sup><br>_nal_|_Ex ante AIT_<sup>_2_</sup><br>_Unconditional_|
|---|---|---|---|---|---|---|
||8-17 years-old|23.0%||-2.6%|******|0.0%|
|||||(0.3%)||(0.5%)|
|Bo|8-11 years-old|6.7%|-1.1%|0.3%||0.0%|
|ys|||(0.8%)|(0.4%)||(0.4%)|
||12-17 years-old|37.3%|-4.7%<br>******|-3.7%|******|0.0%|
||||(2.2%)|(0.5%)||(0.7%)|
||8-17 years-old|9.7%||-0.7%|*****|0.0%|
|||||(0.3%)||(0.3%)|
|Girl|8-11 years-old|4.2%|0.0%|0.4%||0.0%|
|s|||(0.0%)|(0.3%)||(0.4%)|
||12-17 years-old|14.6%|-2.3%<br>*****|-1.3%|*****|0.0%|
||||(1.3%)|(0.5%)||(0.5%)|

_Source_ : Baseline Survey 1997 and Rounds 1-4 – authors' calculation. _Note_ : 1: Results from Skoufias and Parker (2001) - Table 5 (only the results for November 1999 are reported). The coefficients reported are the marginal effects of the _PROGRESA_ program on the probability of children working; 2: Results from simulation on baseline data using BFL model

- ** Significant at 5% level; * Significant at 10% level.

Standard errors and standard deviations in parentheses. For _ex post_ AIT, robust standard errors are shown, which account for clustering of individuals within villages. For _ex ante_ AIT, standard deviations computed by bootstrap method.

22
Table 2 shows the results of a similar exercise comparing _ex ante_ and _ex post_ AIT estimates of the impact of transfers on the proportion of children who are working outside home. The _ex ante_ estimates are again quite close to the _ex post_ SP estimates of impact. Both methods indicate that _PROGRESA_ has the impact of reducing the rate of child labor among 12-17 year olds, but not among 8-11 year olds – quite consistent with what was seen in terms of impact of enrollment using both methods. Among the 12-17 year olds, the program’s impact on child labor is larger for boys than for girls. For both boys and girls of the 12-17 year age group, the _ex ante_ estimates understate the size of the impact relative to the _ex post_ estimates. Consistent with what was seen for enrollment, the _ex ante_ simulations with unconditional transfers show no impact on child labor, suggesting that the conditionality of the transfer was a crucial factor for the impacts.

While it is encouraging to find that the BFL model has good predictive power when compared to results from _ex post_ evaluations with experimental design, the similarity of results may be driven by the restricted and homogenous nature of the baseline sample (of ENCEL) on which the simulations are run. Moreover, from a practical point of view, a specialized baseline survey of program areas may not be available in all cases to conduct _ex ante_ simulations. CCT programs are often set up quickly responding to political needs or to utilize a brief window of opportunity, which may leave little time for baseline data collection. In such cases, a nationally representative household survey, which has much more heterogeneity across households than any baseline survey, is often the only source of data available to conduct simulations on.

Therefore, for methodological as well as practical reasons, it is useful to test the validity of _ex ante_ predictions from the BFL model using the Mexican National Household Survey (ENIGH). This is done by estimating the key parameters of the BFL model ( _αj_ ’s, _M_ and _D_ ) for all children of age 12-17 years with ENIGH 1996 (the round that was the closest to the launch of _PROGRESA_ ), and comparing the estimated parameters with those estimated with a similar model specification for the same age group of children from the ENCEL survey.

The estimated parameters from ENIGH do not differ significantly from those estimated with the baseline survey (Table 3). For all key parameters, the 95 percent confident interval generated by ENIGH and ENCEL overlap each other and can thus be considered statistically similar. However, 23
ENIGH parameters generate smaller program effects compared to the results in Table 1 because all the parameters are a little smaller in size than ENCEL-generated parameter estimates. These results confirm the findings of Leite (2007) for Brazil, which show that estimations based on a more heterogeneous sample such as the one surveyed in the national representative PNAD leads to smaller parameter estimates. By estimating the same set of parameters for a subsample of homogenous households, Leite (2007) found a much higher impact on the enrollment rate and reduction of child work.<sup>30</sup>

**Table 3: Estimating Key Parameters from BFL model** Using Baseline 1997 and Enigh 1996 Surveys

|_Boys and Girls_|_Baseline - ENCEL_|_Enigh_|
|---|---|---|
||_8-17y.o._<br>_12-17y.o._|_12-17y.o._|
|α0|0.025<sup>******</sup><br>0.021<sup>******</sup>|0.018<sup>******</sup>|
||(0.008)<br>(0.008)|(0.009)|
|α1|0.025<sup>******</sup><br>0.021<sup>******</sup>|0.017<sup>******</sup>|
||(0.009)<br>(0.008)|(0.009)|
|α2|0.025<sup>******</sup><br>0.021<sup>******</sup>|0.018<sup>******</sup>|
||(0.009)<br>(0.008)|(0.009)|
|M|53%<sup>******</sup><br>54%<sup>******</sup>|61%<sup>******</sup>|
||(2.02%)<br>(3.09%)|(5.01%)|
|D|65%<sup>******</sup><br>76%<sup>******</sup>|76%<sup>******</sup>|
||(14.8%)<br>(12.2%)|(9.9%)|

_Source_ : Baseline Survey 1997 and ENIGH 1996 _Notes_ :<sup>**</sup> Significant at 5% level;<sup>*</sup> Significant at 10% level. Standard Deviations (in parentheses) computed by bootstrapping.

The BFL model also makes the estimation of program effects on poverty and inequality possible, which is useful to examine the importance of _PROGRESA_ as an anti-poverty program in the short 30 Leite (2007) estimates the set of parameters of interest using a sub-sample of children around the means test threshold. This is done by defining two homogenous groups – potential beneficiaries who are below the means test threshold (treatment) and non-beneficiaries who are above the threshold (control) – who have made their choices subject to similar income constraints. He finds that 65 percent of the movement of children out of the labor market that were already in school can be explained as a response to the Bolsa Escola. These two groups around the means test threshold, according to the Regression Discontinuity Design, are more likely to have similar observable and unobservable characteristics which would make it a good approximation of a randomized experiment.

24
run. This is done by setting 750 pesos as a maximum transfer per household (consistent with the rules of _PROGRESA_ ), using the same poverty line as in Skoufias and Di Maro (2008)<sup>31</sup> and taking into account individual level decisions through the BFL model. The proportion of poor households in the _baseline treatment_ village sample is estimated to be 61 percent, which is the same as in Skoufias and Di Maro. Applying the DID estimator on poverty measures, Skoufias and Di Maro find that between November 1997 and November 1999, _PROGRESA_ was responsible for a 16.5, 24 and 27 percent reduction in the poverty headcount rate, poverty gap and squared poverty gap, respectively, in treatment villages. In comparison to their results, our _ex ante_ simulations suggest a similar impact of _PROGRESA_ on poverty headcount rate (13 percent reduction), but a somewhat larger impact for poverty gap (27 percent reduction) and squared poverty gap (33 percent reduction).<sup>32</sup> Table 4 below shows these results.

**Table 4: Poverty Index: Observed (** **_ex post_ ) and Simulated (** **_ex ante_ )**

||_FGT(_|_0)_|_FGT(_|_1)_|_FGT(_|_2)_|
|---|---|---|---|---|---|---|
||_%_|_σ_|_%_|_σ_|_%_|_σ_|
|Treatment Communities (baseline)|60.9%|1.1%|30.8%|1.1%|21.5%|1.1%|
|Change in poverty_ex post_: Skoufias and Di<br>|||||||
|Maro (2008)<sup>1</sup>|-16.5%|1.6%|-24.3%|1.5%|-29.2%|1.4%|
|Change inpoverty _ex ante_: Simulated<sup>2</sup>|-13.1%||-26.6%||-33.1%||

_Source_ : Baseline Survey 1997 and Rounds 1-4; 1: Skoufias and Di Maro (2008), Table 4; 2: authors' calculation using BFL model _Notes_ : Prices of November 1999 (pesos); poverty line: mean of Nov 98 consumption per capita <u>(taken from Skoufias and Di Maro, 2008)</u> The final step in our analysis is the simulation of the cost of the program based on the BFL model. Table 5 shows that 4,558 out of 7,110 (or 64 percent of) poor families in treatment villages would enroll in the program and accept the condition of school attendance. The average cash transfer for these families in treatment villages is estimated at 303 pesos/month, which is very close to the average transfer of 300 pesos in January 2002 for 1.6 million families observed from official program data. Based on an analytical exercise of scaling up the cost of the program

> 31  The poverty line is equivalent to 176.17 pesos per month (for details, see Skoufias and Di Maro, 2008)

> 32 The _ex ante_ results presented in Table 4 remain the same when the simulations were done _without_ incorporating the conditionality of schooling. This suggests that in terms of meeting the poverty reduction objective of safety net programs, conditionality does not necessarily imply better results. In other words, it suggests that if the main objective of a CCT program is poverty reduction, the transfers are important but the conditions are not – which makes perfect intuitive sense.

25
from the BFL model to the national figure using ENCEL 1997, the total cost of the program is estimated to be around 6.08 billion pesos, which is very close to the actual program cost of 6.06 billion pesos in January 2002 from the official program data (Table 5).

**Table 5: Estimated and Observed** **_PROGRESA_ 's Cost**

||_BFL Simulation_<sup>_1,2_</sup>|_January 2002_<sup>_1,3_</sup>|
|---|---|---|
|Number of Families with Children Receiving Benefits|4,567|1,681,254|
|Average Monthly Transfer for Families with Children|$301|$300|
|Estimated Total Annual Transfer|$16,517,580|$6,061,221,243|
|Scaling Up Total Transfer: Simulated average times<br>families in 2002|$6,080,632,364|n.a.|

_Note_ :

|1: November 1999 prices|
|---|
|2: Estimated by BFL model using Baseline survey of 1997, Rounds 1-4|
|3: National official numbers from:|
|http://www.oportunidades.gob.mx/indicadores_gestion/ene_feb_02/indice.htm|

Thus the BFL model applied to the _PROGRESA_ program in Mexico appears to have strong predictive power for the impact of the program on outcome indicators (i.e. enrollment rate, child labor and poverty) as well as the cost of the program, when compared against relevant benchmarks – namely AIT impacts estimated by _ex post_ evaluations using experimental methods and actual or observed program cost. Since _ex post_ AIT estimates represent a lower bound of the impact on households that actually _received_ treatment (for reasons explained earlier), the _ex ante_ estimates of program impact from BFL should also be seen as a _lower bound_ for the actual impacts of the program on beneficiary households.

# _The impact of Bono de Desarrollo Humano (BDH)_

As before, we compare the _ex ante_ estimates of program impact from the BFL model with _ex post_ findings from Schady and Araújo (2008), henceforth SA. For _ex ante_ simulations, we use the baseline survey of BDH collected in Carchi, Imbabura, Cotopaxi and Tungurahaua and described in section IV above, which is also used by SA as the baseline data in analyzing impact. It is relevant to note that by the authors’ own admission, the results from SA should be treated with some caution as they are not based entirely on random assignment.

26
A summary of descriptive statistics from SA reveals no significant differences between the predefined treated and non-treated samples for a large number of variables (see Appendix, Table A-3 – first two columns), suggesting that the random design of the experiment in terms of the _intended_ treatment and control groups was successful.<sup>33</sup> However, the match between the intended groups with _actual_ BDH recipients is imperfect – transfers went to about 78 and 42 percent of households in the so-called treatment and control groups, respectively. A comparison between _recipients_ and _non-recipients_ of BDH transfer shows significant differences between the two groups for variables like child enrollment, parental education and household size (Appendix, Table A-3, third and fourth columns). Thus selection into the BDH program appears to be non-random.

In order to avoid misspecification due to bias from the contamination of both treatment and control groups, SA computed the “treatment-on-the-treated” or TT estimator using the Two Stage Least Square method. The TT estimator suggests that the BDH increased enrollment rates among compliers by 9.7 percent, with the statistic being only weakly significant (i.e. significant at the 10 percent level but not at 5 percent). Notably, around 25 percent of households in the sample believed that compliance with the condition of school attendance was mandatory for receiving the cash transfer (to be called “conditioned households” henceforth), despite the fact that this conditionality was not enforced by BDH administrators. Out of these conditioned households, 55 percent were from treatment and 45 percent from control groups. Following the re-weighting scheme of Hirano et al (2003), SA reported the Difference-in-Difference or DD estimator, adding a dummy variable for “conditioned” and an interaction term of the “conditioned” dummy variable with the treatment group dummy variable.

As previously mentioned, our BFL model estimator is a type of “Intend-to-treat” estimator, ITT. In this case, it is easy to transform the _ex post_ TT estimator from SA (results from Table 3 in SA) for comparison with our estimator, as follows:

> 33 Children from the intended treatment and control groups are indistinguishable in the baseline data in terms of their enrollment rate, grade attainment, labor participation, gender, per capita expenditure, assets, parental education and household size,

27
Another issue to take into account is the amount of transfer to be allocated to each child. Our BFL model is based on the assumption that the decision to enroll (or not) every child in school is made (by the child or the household) independently, taking into account the transfer amount for the individual child. But the BDH program transfers US$15 per household, independent of the number of enrolled children in the household, which makes the decision of sending or keeping every child in school essentially a household level decision.

On the one hand, if we set $15 as the direct transfer amount, we suppose that each child will add to its utility an amount of _$15.αj_ (for all j= 1, 2),<sup>34</sup> which will lead to an overestimation of the effects of the program. This can then be considered as the upper-bound of the estimated impact of BDH from the BFL model. On the other hand, the BFL model does allow us to construct alternative scenarios about how the $15 is distributed within the household, namely the size of the de facto transfer to every child when there are multiple children in the household. We propose three alternative scenarios. The first scenario is one where the transfer received by each child is equal to $15 divided by the number of children aged 6-17 years in the household, with a minimum of $3 per child. In the second scenario, we set an individual transfer of US$5 because the average number of potential beneficiaries is close to three per household. In the third scenario, the individual transfer is set at $7.5. In order to account for the cost of the program and average size of transfer per household, we enforce the official household maximum of $15 as defined by the BDH program and also compute its impact _without_ any household maximum.

Table 6 shows all _ex post_ estimators obtained by SA and the results of the different scenarios proposed using the BFL model. As expected, the _ex ante_ or simulated ITT estimator under the assumption of US$15 transfer per child (IIT<sup>1</sup> ), at 14.58 percent, is almost double the size of the _ex post_ impact measured by SA at 10 percent. The annual cost of the program under scenario 1 (US$15 per household, despite individual decision) is US$166,000 if we set the household ceiling at US$15. Allowing for different amounts of transfer per household would increase the average transfer to US$32.33 and the total annual cost would more than double. However, the effect on enrolment rate generated by scenario 1 is unlikely, as a child living with its siblings will never make a decision on program participation based on a US$15 transfer.

> 34 Refer to Equation (4) earlier.

28
The next three proposed scenarios have lower estimated impacts ranging from 5.86 percent to 9 percent; and for all three cases, the _ex post_ ITT estimator falls in the 95 percent confidence interval of the _ex ante_ ITT estimator. For these scenarios, the average transfer per household ranges from US$9.8 to US$15.95, regardless of whether the US$15 ceiling is taken into consideration. As a consequence, the estimated cost of the program never extrapolates to the cost in scenario 1 of US$15 per household. For example, this result suggests that we would have observed higher program effects (without affecting the overall cost of the program) by making individual, smaller transfers of US$7.5 per child, so that the amount of transfer per household ranges from US$7.5 to US$45 depending on the size of the household.

For the reasons described earlier, we cannot calculate _ex post_ results for the BDH program in a way that would allow for a direct comparison with _ex ante_ predictions as we have done for the case of _PROGRESA_ . However, the prediction power of the BFL model is reasonable and consistent with our expectations conditional on the assumption made about the allocation of the household transfer among beneficiaries. As we would expect, considering each child in the sample to be a household under the BFL definition (and receiving US$15) overestimates the impact of the BDH. Keeping the average transfer per household close to aor above US$15 and using smaller amounts of transfer per child, the BFL model estimates program effects on enrollment rate that are not statistically different (at 5 percent level of significance) from _ex post_ results reported by SA for all simulated ITT estimators.

# **VI. Conclusion**

The results of our paper suggest that _ex ante_ and _ex post_ methods for estimating impacts can effectively complement each other to inform the design and evaluation of social programs. In principle, combining an _ex ante_ simulation exercise using a structural model of schooling choices with an experimental evaluation design would help policymakers to assess the implications of alternative choices for program design, potential impacts and cost implications upfront, and rely on _ex post_ evaluation results to validate and identify the actual impacts of the program. For _ex ante_ models to be used this way, however, their results need to be credible in terms of accuracy or predictive power. Our findings make some progress in this direction. Our results suggest that _ex ante_ predictions of certain impacts of CCT programs using the BFL model match up well with 29
_ex post_ “gold standard” results from experiments, which should lend confidence to policymakers about the reliability of _ex ante_ estimates from this model.

The primary analysis of the paper consists of comparing the _ex ante_ simulated impact on enrolment rates using the BFL model with the _ex post_ impact estimates from a randomized experiment. For the case of the _PROGRESA_ program in Mexico, we can also compare the _ex ante_ and _ex post_ estimates of program impact on poverty. Such a comparison of methods is also a test to validate the hypothesis central to such _ex ante_ models: cross-sectional income effects, estimated on the basis of the representative household sample, coincide with the income effects generated by a program.

For _PROGRESA_ , we estimate that the predicted impact tends to understate but is not significantly different (at 5 percent level of statistical significance) from _ex post_ “intent to treat” or ITT estimators of impact for all age groups of boys and girls. Moreover, the BFL model estimates similar results in terms of changes in poverty levels. For the Ecuadorian program _Bono de Desarrollo Humano_ (BDH), the use of the BFL model, with reasonable assumptions about how the household level transfers would translate to de facto transfers for every child, yields simulated impacts that are again statistically similar to _ex post_ ITT impact estimates. The fact that _ex ante_ predictions from the BFL model are validated by _ex post_ estimates also lends confidence to the use of the model for analyzing other types of questions related to program design. These are questions that are important for the program designer before designing the program, but too expensive or cumbersome to implement as a pilot to evaluate the actual impacts _ex post_ . An example of such questions is: What would have been the likely impact of _PROGRESA_ on enrollments if it were an _unconditional_ cash transfer? Using the BFL model we find that transfers without conditions would _not_ have produced the enrollment impacts seen from the conditional transfer. In case of BDH, the _ex ante_ simulations suggest that program impact on enrollment rates could have been higher if the transfers were smaller and for each individual child (instead of a flat amount every household), for an unchanged overall cost of the program.

Thus the _ex ante_ method proposed by Bourguignon, Ferreira and Leite (2003) can be a powerful simulation method in designing or reforming CCT programs that aim to improve enrollments. The built-in advantages of an _ex ante_ approach – in terms of cost, practicality and the stage of 30
the program cycle when it can be done – implies that a model that performs well against the benchmark of “gold-standard” _ex post_ evaluations can be extremely helpful to inform policymakers about program design. One major advantage is that in many cases, such a model can allow for comparison between the likely impacts of alternative ways of designing a program – without the time, cost and logistics involved in setting up multiple pilot experiments to obtain _ex post_ evaluation results. There are important caveats as well to the use of the BFL model, the most important among which is that _ex ante_ analysis cannot incorporate any _unanticipated_ changes that may occur during the process of program implementation, be that in the implementation process or in any other extraneous events (including large shocks) that can affect the behavior of program participants. The _ex ante_ exercise would then need to be followed up with an _ex post_ impact evaluation that is able to capture the _actual_ effect of the program on the outcomes of interest, taking into account all the possible changes in the program and the environment it operates in that could have occurred as it was being implemented.

31
**Table 6: Children’s Occupational Choices in Ecuador 2003-2005** Actual and Counterfactual Enrollment Rate for Target Population

|||TT|**_Ex post_**<br>Probability of<br>receiving benefits if|ITT||**_Ex ante_**|||
|---|---|---|---|---|---|---|---|---|
||d1|(a)|eligible<br>(b)|(c)=(a)(b)|Simulated ITT<sup>1</sup>|Simulated ITT<sup>2</sup>|Simulated ITT<sup>3</sup>|Simulated ITT<sup>4</sup>|
|Table 4 - Results|||||||||
|2SLS|10.00%|10.00%|77%|7.7%|14.58%|7.63%|5.86%|9.00%|
||4.72%|4.72%|||0.53%|0.82%|1.10%|0.60%|
||Average Tran|sfer per House|hold with Ceiling of $15|per household|15.00|11.91|9.80|12.31|
||||Number|of households|645|631|596|603|
|||||Annual Cost|116,100.00|90,192.00|70,080.00|89,100.00|
||Avera|ge Transfer per|Household without Hou|sehold Ceiling|32.33|11.92|10.35|15.95|
||||Number|of households|645|631|599|605|
|||||Annual Cost|250,200.00|90,264.00|74,400.00|115,830.00|

Source: Schady and Araújo (2008) table 4 column (3).

Note:

- 1 - R$15 per child assuming unitary child household.

- 2 - $15 divided by number of children age 6-17 in the household. Minimum transfer $3 per child.

- 3 - $5 per child.

- 4 - $7.5 per child.

- ** Significant at 5% level; * Significant at 10% level.

Standard Deviation computed by bootstrap method.

32