metadata: authors: 0: author: Elbers, Chris
1: author: Gunning, Jan Willem
count: Vietnam
docna: 0: docna: Evaluation of development programs :
            randomized controlled trials or regressions ?
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Public Sector Management and Reform,Health Economics & Finance,Educational Sciences,HIV AIDS,Health Insurance,Cholera,Communicable Diseases,Leprosy
teratopic: Science and Technology Development,Poverty Reduction,Health, Nutrition and Population,Education
abstracts: cdata!: Can project evaluation methods be used
            to evaluate programs: complex interventions involving
            multiple activities? A program evaluation cannot be based
            simply on separate evaluations of its components if
            interactions between the activities are important. In this
            paper a measure is proposed, the total program effect (TPE),
            which is an extension of the average treatment effect on the
            treated (ATET). It explicitly takes into account that in the
            real world (with heterogeneous treatment effects) individual
            treatment effects and program assignment are often
            correlated. The TPE can also deal with the common situation
            in which such a correlation is the result of decisions on
            (intended) program participation not being taken centrally.
            In this context RCTs are less suitable even for the simplest
            interventions. The TPE can be estimated by applying
            regression techniques to observational data from a
            representative sample from the targeted population. The
            approach is illustrated with an evaluation of a health
            insurance program in Vietnam.
geo_reg_key: 80475,80451,80471
repnme: repnme: Evaluation of development programs :
            randomized controlled trials or regressions ?
display_title: Evaluation of development programs :
            randomized controlled trials or regressions
listing_relative_url: /research/2013/09/18192682/evaluation-development-programs-randomized-controlled-trials-or-regressions
topicv3: Human Resources m1326304 1211,Education m1327048 470,HIV/AIDS m1327522 1416,Health Insurance m1327759 1287,Health m1504661 3480
publishtoextweb_dt: 2013-09-01T00:00:00Z
historic_topic: Science and Technology Development,Poverty Reduction,Health, Nutrition and Population,Education
no_of_pages: 31
model_extractions: WPS6587 Policy Research Working Paper 6587 Evaluation of Development Programs Randomized Controlled Trials or Regressions?

_Chris Elbers Jan Willem Gunning_ The World Bank Development Economics Vice Presidency Partnerships, Capacity Building Unit September 2013
Policy Research Working Paper 6587

# **Abstract**

Can project evaluation methods be used to evaluate programs: complex interventions involving multiple activities? A program evaluation cannot be based simply on separate evaluations of its components if interactions between the activities are important. In this paper a measure is proposed, the total program effect (TPE), which is an extension of the average treatment effect on the treated (ATET). It explicitly takes into account that in the real world (with heterogeneous treatment effects) individual treatment effects and program assignment are often correlated. The TPE can also deal with the common situation in which such a correlation is the result of decisions on (intended) program participation not being taken centrally. In this context RCTs are less suitable even for the simplest interventions.

The TPE can be estimated by applying regression techniques to observational data from a representative sample from the targeted population. The approach is illustrated with an evaluation of a health insurance program in Vietnam.

This paper is a product of the Partnerships, Capacity Building Unit, Development Economics Vice Presidency. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at c.t.m.elbers@vu.nl and j.w.gunning@vu.nl.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Evaluation of Development Programs:**

# **Randomized Controlled Trials or Regressions?**

_Chris Elbers and Jan Willem Gunning*_

## **JEL Classification Codes:** C21, C33, O22

**Keywords:** program evaluation; randomized controlled trials; policy evaluation; treatment heterogeneity; budget support; sector-wide programs; aid effectiveness

## **Sector Board: Economic Policy (EPOL)**

* The authors are professors at the VU University Amsterdam and fellows of the Tinbergen Institute. Their addresses are c.t.m.elbers@vu.nl and j.w.gunning@vu.nl (corresponding author). They are grateful to Remco Oostendorp, Menno Pradhan, Martin Ravallion, Elisabeth Sadoulet, Finn Tarp, the editors and two anonymous referees of the _World Bank Economic Review_ and to seminar participants in Amsterdam, Namur, Oxford and Paris for very valuable comments on previous versions.
Experimental methods for impact evaluation presuppose that the intervention is well-defined: the “project” is limited in space and scope (e.g. Duflo _et al._ , 2008). However, governments, NGOs and donor agencies are often interested in evaluating the effect of a program consisting of various interventions, e.g. a sector-wide health or education programs (De Kemp _et al._ , 2011). Program evaluation faces two complications. First, a sharp distinction between treatment and control groups is usually impossible. For example, a program in the education sector may involve activities such as school building, teacher training and supply of textbooks. Typically _all_ communities are affected in some way by the program, but they may differ dramatically in what interventions they are exposed to and the extent of that exposure. Secondly, in a program the interventions are typically implemented at various administrative levels so that the policy maker has only imperfect control over actual treatment.

The impact of such a program cannot simply be calculated on the basis of the results of randomized controlled trials (RCTs). This would run into well known problems of external validity (Bracht and Glass, 1968, Rodrik, 2008, Ravallion, 2009, Banerjee and Duflo, 2009, Deaton, 2010, Imbens, 2010) even if the program involved only a single intervention. In addition, if the program involves multiple interventions and interactions are important then it is not clear how RCT evaluations of individual components of the program should be combined to an overall assessment of the program. However, regression techniques can be used for program evaluation. This involves drawing a representative sample of beneficiaries (e.g. households, schools, communities) and collecting data on the combination of interventions experienced by each beneficiary, together with other possible determinants of the outcome variables of interest. Regression techniques can then be used to estimate the impact of the various interventions.<sup>1</sup> In this paper this approach is generalized by allowing for treatment heterogeneity and a way of estimating aggregate program impact is proposed.

2
Obviously, the intervention variables are likely to be endogenous in a regression analysis. For example, an unobserved variable such as the political preferences of the community may affect both the impact variable of interest and the intervention. Also, the impact of the intervention will differ between beneficiaries and the allocation of interventions across beneficiaries may be based on such treatment heterogeneity, either through self-selection or through the allocation decisions of program officers. Heckman (1997) and Heckman _et al._ (2008) call this “selection on the gain”. The first complication is usually dealt with by using panel data or by randomized assignment of treatment. The second complication is much more serious. It may be particularly hard for RCTs when program assignment in practice cannot be mimicked by assignment to the treatment arm in an RCT since this would not capture the way program officers take their decisions. However, it will be shown that regression techniques can be adapted so as to produce an appropriate estimate of the program effect.

The paper is organized as follows. In the first section the total program effect (TPE) is introduced. This measure extends the average treatment effect on the treated (ATET). The TPE is suitable for complex interventions and can deal with selection on the gain (treatment heterogeneity). Then two complications are considered: correlation between program variables and the controls in section 2 and spillover effects in section 3. Section 4 investigates whether estimating the TPE using RCTs is an alternative. The approach is illustrated in section 5 by estimating the TPE for a health insurance intervention in Vietnam. Section 6 concludes.

# **_I. The Total Program Effect (TPE)_**

Consider the following model: 3
where _y_ measures an outcome of interest, in this paper taken to be a scalar; _t =_ 0, 1 is the time of measurement; and _i_ = 1,..., _n_ denotes the unit of observation, _e.g._ households or locations. _P_ denotes a vector of the interventions to be evaluated and _X_ a vector of observed controls.<sup>2</sup> The _P_ -variables can either be binary variables or multi-valued (discrete or continuous) variables. α and<sup>β</sup> _i_<sup>are vectors of parameters,γ</sup> _t_<sup>denotes a time effect andη</sup> _i_<sup>represents time-invariant</sup> unobserved characteristics and ε _it_ is the error term, assumed to be independent over time. It is also assumed that the interventions and control variables are uncorrelated with the error process: At this stage _P_ and _X_ are assumed to be independent: This will be relaxed in section 2. Note that equation (1) excludes spillover effects of the type where<sup>_y_</sup> _it_<sup>depends on</sup> _Pjt_ ( _i_ ≠ _j_ ) and _j_ is not necessarily included in the sample. This point will be discussed in section 3. In many applications (1) will represent a reduced form or “black box” regression, but it can also represent a structural model.

The evaluator is interested in the expectation (in the population) of the effect of interventions on the outcome variable, the total program effect (TPE):<sup>3</sup> Note that the impact parameters<sup>β</sup> _i_<sup>need not be the same for all</sup><sup>_i_: heterogeneity of program</sup> impact is allowed.

As an example consider a very simple special case: 4
where<sup>_P_</sup> _it_<sup>now is a binary variable rather than a vector,</sup> _Pi_ 0 = 0 for all and _i Pi_ = _Pi_ 1 − _Pi_ . Taking first differences gives: where γ = γ 1 −γ 0<sup>.</sup> This is analogous to the equation for a standard project evaluation, but written in differences.<sup>4</sup> The TPE for this case equals _E_ β _iPi_ which is related to the familiar average treatment effect on the treated (ATET) In another special case of equation (1) the TPE can be identified as follows. Assume that data are available from a random sample and that for a subsample (the “control group”) there is no change in the interventions: _Pi_ 1 = _Pi_ 0<sup>.</sup> (At this stage it is not assumed that the assignment to intended “treatment” and “control” groups is random.) Taking first differences in (1) for this group gives: This allows estimation of α and hence α∆ ˆ _X i_ so that the TPE can be estimated as However, in a program consisting of multiple interventions, the context of this paper, there will usually not be a sufficiently large control group to make this identification strategy realistic.

Indeed, typically the control group will be empty: all _i_ will have experienced a change in at least some components of the vector ∆<sup>_P_</sup> _i_ .

For this more general case 5
Allowing for “selection on the gain”, correlation between impact parameters<sup>β</sup> _i_<sup>and the</sup> program variables<sup>_P_</sup> _i_<sup>and also for correlation betweenβ</sup> _i_<sup>and</sup> _X i_ equation (3) can be rewritten as where ω _i_ = ∆ε _i_ + ( β _i_ − _E_ ( β _i_ | ∆ _X i_ , ∆ _Pi_ )) ∆ _Pi_ and this is uncorrelated with ∆ _X i_ and ∆ _Pi_ . The term _E_ ( β _i_ | ∆ _X i_ , ∆ _Pi_ ) can be approximated linearly:<sup>5</sup>

Substitution in (4) and collecting terms gives where is the approximation of _Ti_ = _E_ ( β _i_ ∆ _Pi_ | ∆ _X i_ , ∆ _Pi_ ).

Equation (5) can be estimated using the sample data. The estimated coefficients can then be used to estimate<sup>_T_</sup> _i_<sup>as</sup> The TPE can now be estimated as the average of<sup>_T_ˆ</sup> _i_<sup>in the sample.</sup> where bars denote sample averages.<sup>6</sup> 6
In practice this means that one regresses ∆<sup>_y_</sup> _i_ on ∆ _X i_ , ∆<sup>_P_</sup> _i_<sup>and their interactions with</sup> ∆<sup>_P_</sup> _i_ and collects all terms involving ∆<sup>_P_</sup> _i_ to calculate the total program effect. Since the estimated TPE is linear in the θ<sup>ˆ</sup> parameters its standard error can be obtained from the covariance matrix of the OLS-coefficients.

It is instructive to consider the special case of equation (5) where _Di_ = ∆ _Pi_ is a binary variable taking the value 1 for the treatment group and 0 for the control group, i.e. the case of a difference-in-difference analysis. Equation (5) now reduces to 2 since in this case _Di_ = _Di_ . Compared to a standard diff-in-diff regression this equation contains the interaction term _Di_ ∆ _X i_ .

The total program effect will in this case be estimated as This shows that when the sample is representative sample means can be used to construct the total program effect. The interaction term in (7) avoids the bias resulting from correlations between treatment effects and either program participation or controls.

Many diff-in-diff studies do not include the interaction terms (e.g., Khandker _et al_ ., 2009 or Almeida and Galasso, 2010). Studies that do often report estimates of impact for different values of the controls _X_ which makes it difficult to assess the aggregate impact of a program.

7
Equation (1) allows for two types of selection effects:<sup>_P_</sup> _it_<sup>may be correlated withβ</sup> _i_<sup>or with the</sup> unobserved characteristics η _i_ . A correlation of<sup>_P_</sup> _it_<sup>and</sup> η _i_ is dealt with by differencing, as in (3).<sup>7</sup> However, the TPE measures the effect of the program _inclusive_ of selectivity in the assignment of program interventions resulting in a correlation of<sup>β</sup> _i_<sup>and</sup> ∆<sup>_P_</sup> _i_ . This is appropriate since the way the program was assigned (in an _ex post_ evaluation) or will be assigned (in an _ex ante_ evaluation) is one of its characteristics. If the program was successful in part because program officers made sure the program interventions were assigned to households or locations where they expected a high impact, then obviously the evaluation should reflect this. In fact the evaluation would be misleading if it tried to “correct” for such selection effects by presenting (if this were feasible) an estimate ( _E_ β _i_ ) of the program’s impact if it had been assigned randomly.

Recall that in the special, binary case of a `project’ evaluation TPE = _E_ β _i_ ∆ _Pi_ = ATET × _E_ ∆ _Pi_ . If administrative data can be used to estimate _E_ ∆ _Pi_ the question arises whether the ATET is identified in an RCT. Obviously this is the case if<sup>β</sup> _i_ = β for all _i_ . More generally, if ∆<sup>_P_</sup> _i_<sup>andare independent the TPE can be estimated on the basis of an RCT: the trial would give anβ</sup> _i_ estimate of _E_ β _i_ which in this case is also the ATET. A special case of independence is that of universal treatment (<sup>_P_</sup> _i_ = 1 for all _i_ ).<sup>8</sup> In the most general case when ∆<sup>_P_</sup> _i_<sup>andβ</sup> _i_<sup>are not</sup> independent the ATET as established by an RCT may differ from the ATET in the population and estimating the TPE on the basis of RCTs can become problematic. This issue will be considered in section 4.

8
# **_II. Correlation between P and X_**

In the previous section _P_ and _X_ were assumed to be independent. ( _P, X_ ) correlations are often important in evaluations. For example, changes in teacher training may induce changes in parental input.<sup>910</sup> Not all such inputs will be observed (e.g. additional parental help with homework will probably not be recorded);<sup>_P_</sup> _it_<sup>will then be correlated withβ</sup> _i_<sup>and this was</sup> already considered in the previous section. Conversely, if the parental input is observed then<sup>_P_</sup> _it_ will be correlated with _X it_ . In that case the TPE identifies the direct effect of _P_ , but not its total effect (including the indirect effect through induced changes in _X_ ). If the induced effect is to be included then the affected components of ∆ _X i_<sup>should be omitted from the regression (5).</sup> If causality is in the reverse direction, from ∆ _X i_ to ∆<sup>_P_</sup> _i ,_ then there is no need to amend the section 1 estimate of the TPE since there is no induced change in ∆ _X i_ . (The asymmetry arises because in either case the interest is in the impact of changes in ∆<sup>_P_</sup> _i_ , rather than in the impact of changes in ∆ _X i_ .) In the general case where the direction of causality is not known it will usually not be possible to estimate the indirect effect of the program. Occasionally, however, appropriate instruments can be found so that the impact of ∆<sup>_P_</sup> _i_ on ∆ _X i_ can be identified.

9
# **_III. Spillover Effects_**

Recall that in section 1 spillover effects were excluded: in equation (1)<sup>_y_</sup> _i_<sup>of case</sup><sup>_i_does not</sup> depend on<sup>_P_</sup> _j_<sup>of case</sup><sup>_j_. In evaluations there are two important situations where this assumption</sup> is untenable. First, Chen _et al._ (2009) and Deaton (2010) discuss the possibility that policy in control villages is partly determined by policies in treatment villages so that the SUTVA (stable unit treatment value assumption) is violated. Indeed, if policies thus affected are not represented in the policy vector<sup>_P_</sup> _i_<sup>this creates a classical case of omitted variable bias. In Chen</sup><sup>_et al._the</sup> problem arises because the data record participation in a particular program as a binary<sup>_P_</sup> _i_ variable, while other programs which may affect the outcome are initially ignored. In the approach advocated in the present paper all potentially relevant programs would in principle be included in<sup>_P_</sup> _i_<sup>so that the problem of SUTVA violation is avoided.11 Secondly, policies in</sup> village _j_ may affect outcomes in village _i._ For example, a program aimed at an infectious disease in village _j_ may affect health outcomes in the “untreated” village _i._<sup>_12_</sup> If the external effects of policy are general equilibrium effects such as regional wage increases, it will be hard to identify the full impact of a policy. But often more structure can be imposed, e.g. by including a proxy for relevant policies in neighboring villages in the outcome regression, so that equation (3) is extended to where ∆ _Ki_ is the proxy for policy changes in the neighborhood. If there is sufficient variation in _Ki_ then δ is identified in this regression. The TPE would then be _E_ β _i_ ∆ _Pi_ + δ _E_ ∆ _Ki_ .

10
# **_IV. Regression Methods and RCTs Compared_**

In section 1 it was shown how the TPE can be estimated using regression methods. A natural question is whether the TPE can also be estimated using RCTs. Using RCTs may be difficult, e.g. because in programs the distinction between treatment and control groups may break down. However, there may be problems even in the case of binary treatments, namely under treatment heterogeneity when the probability of treatment is correlated with the individual impact parameters<sup>β</sup> _i_<sup>and unknown to the evaluator. If this correlation arises through self-selection then</sup> the usual response is to consider the average treatment effect on the treated rather than the average treatment effect in the population. If, however, the correlation arises at a higher level, e.g. because the policy maker targets on observables, then an RCT would have to mimic this assignment, possibly by stratifying the sample on the basis of the targeting variables.

But in many government and NGO programs the “policy maker” does not directly control the _P_ variables: assignment is decided by lower level staff (“program officers”) on the basis of private information, variables that cannot be observed by the policy maker or the evaluator. In this case an RCT can still identify the TPE, but at the cost of having to randomize at a higher level than the treatment under consideration: randomization would apply to program officers rather than beneficiaries. This implies that the power of the statistical analysis may be reduced. It also involves losing the direct link with the intervention.

This may be illustrated with an example. Consider the following model where<sup>β</sup> _i_<sup>andε</sup> _i_<sup>are independent,</sup><sup>_P_</sup> _i_<sup>is binary and</sup> _E_ ε _i_ = 0 . For simplicity<sup>β</sup> _i_<sup>will be considered</sup> as the intention-to-treat impact, so that a subject _i_ ’s refusal to undergo offered treatment<sup>_P_</sup> _i_<sup>is</sup> 11
reflected in<sup>β</sup> _i_<sup>, rather than in</sup><sup>_P_</sup> _i_<sup>. Program implementation involves program officers who have</sup> imperfect knowledge of<sup>β</sup> _i_<sup>: they perceive</sup> ω _i_ = β _i_ +η _i_ and will assign treatment if and only if ω _i_ > 0 . Assume that<sup>η</sup> _i_<sup>has mean zero and is independent ofβ</sup> _i_<sup>andε</sup> _i_<sup>. Crucially, this</sup> knowledge of program officers is unknown to the evaluator. Denote the CDF of<sup>η</sup> _i_<sup>by</sup><sup>_F_. With</sup> this assignment rule<sup>_P_</sup> _i_<sup>is exogenous (</sup><sup>_i.e._independent of</sup> ε _ij_ ).

An RCT evaluation might involve drawing a random sample from the population and assign treatment randomly within this sample. The researcher would then estimate the program’s intention to treat effect (ITE) as _E_ β _i_ . The TPE would be estimated as _E_ β _iEPi_ .

This would be incorrect since, under the assumptions made above TPE = _E_ β _iPi_ = _E_ ( β _i_ | β _i_ +η _i_ > 0) _P_ ( β _i_ +η _i_ > 0) = _E_ [(1 − _F_ ( −β _i_ )) β _i_ ] ≠ _EPEi_ β _i_ . (Note that _E_ (1 − _F_ ( −β _i_ )) = _P_ ( β _i_ +η _i_ > 0) = _EPi_ . As before, the ATET = TPE / _EPi_ .) The problem arises because in this case the RCT design does not mimic the actual assignment process. To obtain an unbiased estimate of the TPE randomization would have to take place at a higher level, that of the program officers.<sup>13</sup> The control group then consist of program officers who never “treat” and the treatment group of program officers who sometimes (but not always) treat.

The proposed regression method gives an unbiased estimator of the TPE using observational data for ( _yi_ , _Pi_ ) from a random sample of the population. The difference is that while the RCT approach compares average outcomes at the level of program officers the regression approach does so at the level of beneficiaries. The RCT approach therefore has lower statistical power.<sup>14</sup> 12
Moving beyond the example there is a more fundamental objection to the RCT approach if outcomes depend not only on _P_ but also on _X_ , as in (1). If the RCT involved randomization over actual program officers then it is unlikely that randomization can also be achieved in terms of all the confounding _X_ variables since program officers will not have been posted randomly across space. This introduces a correlation between _X_ and characteristics of the program officers and hence a correlation between _P_ and _X_ . The two groups of program officers (“treatment” and “control”) will therefore differ systematically so that internal validity is lost.<sup>15</sup> The proposed approach, by contrast, collects data at the level of beneficiaries and can therefore control for differences in _X_ .

In summary, estimating the TPE on the basis of group averages from RCTs becomes problematic when _β_ and _P_ are correlated as a result of targeting on the basis of unobservables. If one randomizes at the level of beneficiaries the TPE estimator will be biased because the correlation is not taken into account. If one randomizes at the level of program officers the estimator is inefficient and, if confounders are important, may become inconsistent.

# **_V. An Empirical Example: Estimating the Total Program Effect for a Health Insurance_**

# **_Program in Vietnam_**

To illustrate how the total program effect can deviate from a naïve approach to calculating the effect of a program a study of the impact of a health insurance program in Vietnam (Wagstaff and Pradhan, 2005) is reconsidered. Health insurance was introduced between the 1992-93 and the 1997-98 rounds of the Vietnam Household Living Standards Survey (General Statistics Office of Vietnam, 1993 and 1998). To account for possible treatment heterogeneity Wagstaff and Pradhan match households on propensity scores and then compare changes in health 13
outcomes (as well as some non-health outcomes) between insured households or individuals and (matching) uninsured households or individuals. They find modest favorable effects on children’s nutritional status, a mild effect on health expenditure and a sizeable effect on nonhealth spending.

A propensity score based approach is not suitable for calculation of a total program effect since the common support requirement in a PSM approach will exclude part of the population in a systematic way. Therefore the Vietnam data are used to estimate the effect of the program using a standard diff-in-diff approach _i.e._ without allowing for heterogeneity (labeled ‘naïve’). The results are compared with an estimate of the TPE. In this case the ‘program’ is a simple intervention.<sup>16</sup> This makes a comparison with a standard approach clearer.

The data are summarized in Table 1. A difficulty is that some of the outcome variables are individual anthropometric measurements while only households can be matched between survey rounds. Therefore the individual measurements have been averaged per household - a crude procedure only suitable for the current purpose of illustrating the TPE. Lacking information on 1992-3 the sampling weights from 1997-8 are used; clustering is also based on the 1997-8 survey round.

The outcome variables considered are changes in arm circumference, height, body weight, health expenditure and total expenditure. The explanatory variables are the other variables shown in Table 1 (insurance status and the controls school attended, currently attending school, gender, age, a farm dummy, household size): and their interaction with the intervention variable are used as explanatory variables.<sup>1</sup> When total expenditure is not a dependent variable it is also used as control variable.

14
Table 2 summarizes the results. First a naïve regression is run (without interaction terms) and the implied program effect (calculated as the regression coefficient of insurance times mean insurance). This naïve program effect is then compared with a TPE calculated as in equation (6).

The results show striking differences between the two methods. In the case of arm circumference the standard method would have led to the conclusion that insurance had no (significant) effect. Once treatment heterogeneity is allowed for the effect is in fact highly significant albeit very small. For height neither method finds a significant effect. For body weight both methods show a significant increase but the effect is more than twice as large when heterogeneity is allowed for.

Insurance appears to have no significant effect on health expenditure irrespective of the method used. Both methods do find a substantial (and significant) effect of insurance on total consumption. Again, the effect is stronger once one takes heterogeneity into account.

Obviously, there is no reason why these results should generalize. However, they do suggest that treatment heterogeneity can have a substantial effect on the estimates of a program’s impact. A simple way to investigate this possibility is to test for the joint significance of the coefficients on the variables which would not normally be included in the regression: the interactions of treatment variables with themselves and with the controls. When this test indicates that heterogeneity may be an issue it is advisable to calculate the TPE.

# **_VI.  Conclusion_**

Policy makers in developing countries, NGOs and donor agencies are under increasing pressure to demonstrate the effectiveness of their program activities. At the same time there is a growing 15
interest in using randomized controlled trials (RCTs) for impact evaluation of projects. This raises the question to what extent RCTs can be used to evaluate programs, for instance by aggregating the impact of the components of the program. This question is particularly relevant for the evaluation of budget support or of NGOs which typically involve a wide variety of activities.

The strength of RCTs is in establishing proof of principle. Going further and using RCTs to estimate the impact of programs is possible in special cases but becomes problematic if the probability of assignment is correlated with the effectiveness of the intervention. For example, teachers may give more attention to children who they think can benefit more from it. An RCT which randomizes at the level of beneficiaries (children) would produce a biased estimate of the program effect by ignoring this correlation between assignment and treatment effects.

Alternatively, randomization at the appropriate level (teachers) would require a larger sample for the same precision. If confounders are important and correlated with characteristics of the program officers, the RCT-based estimate of the program’s impact would even be inconsistent.

The approach proposed in this paper requires observational panel data for a representative sample of beneficiaries rather than experimental data for randomly selected treatment and control groups. If treatment is exogenous this will correctly reflect the assignment process even under treatment heterogeneity. Instead of estimating average impact coefficients for each of the various interventions of the program, the expected value (across beneficiaries) of the total impact of the combined interventions is estimated. This gives the total program effect (TPE). The paper has shown how and under what conditions regression techniques can be used to estimate the TPE in the presence of selection effects. As an example TPE estimates for a simple intervention: a health insurance program in Vietnam were presented. The example shows that 16
allowing for heterogeneity can lead to very different estimates of a program’s effect. The proposed method offers a simple way of dealing with such heterogeneity.

The approach has three advantages. First, by using observational data for a random sample from the population of intended beneficiaries external validity is ensured. While the disadvantages of observational data are well known, this is an important advantage. Secondly, by focusing on the combined effect of program components they are automatically correctly weighted. Finally, it avoids the problems which RCTs encounter when assignment is imperfectly controlled and correlated with unobservables, as is plausible in development programs.

17
# _Notes_

> 1 This approach is discussed in White (2006) and Elbers _et al._ (2009).

> 2 Here _P_ reflects “actual” treatment . In principle it could reflect “intended” treatment if intended treatment can be observed, e.g. because intended beneficiaries were offered vouchers.

> 3 Strictly speaking this is the total effect of _changes_ in the program. The symbol _E_ is used for population averages and a bar over a variable for sample averages. Note that the total program effect does not include general equilibrium effects of the program.

> 4 This assumes that the autonomous trend γ = γ 1 −γ 0 is the same for all subjects (or, alternatively that the

difference ∆<sup>γ</sup> _it_ is exogenous and can be treated as part of the residual). In the terminology of double differencing this is the assumption of parallel trends. If this assumption is questionable then data for more periods are needed to estimate how trends depend on _P_ . This paper abstracts from this complication and limit the analysis to two periods. The extension to more periods is non-trivial but conceptually straightforward.

5 Higher order approximations would not change the argument but it should be noted that the number of regressors expands very rapidly. De Janvry _et al._ (2012) account for treatment heterogeneity in a similar way in the context of a schooling program.

> 6 Obviously, to identify θ 4 a restriction on parameters like θ 4, _k_  = θ 4,  _k_<sup>is required.</sup>

7 Differencing is sufficient because of the assumption of parallel trends (cf. footnote 5).

> 8 Imbens (2010) describes a reduction in class size in _all_ California schools. This is an example of universal treatment.

9 Deaton (2010) gives the example where random assignments made by the central government (e.g. the Ministry of Education) are partly offset by induced changes in allocations by local or provincial governments. Ravallion (2012) gives a similar example and Chen _et al._ (2009) quantify such a spillover effect in China. Similarly, the political economy may be such that the central government is unable to prevent allocations being diverted to favored ethnic or political groups. In either case _Pi_ might be correlated with _βi_ .

> 10 This is similar to the case considered by Das _et al._ (2004, 2007) where teacher absenteeism as a result of HIV/AIDS induces greater parental input.

11 Recall that the approach does not involve a distinction between treatment and control groups: most if not all subjects receive some treatment.

18
12 This has implications for sampling: since data on policies in neighboring villages are required one must sample groups (possibly pairs) of adjacent villages.

> 13 Duflo _et al._ (2008, pp. 3935-37) make this point in a similar context (partial compliance) concluding that “One must compare _all_ those initially allocated to the treatment group to _all_ those initially randomized to the comparison group”.

14 This is shown in the supplemental appendix.

15 This is shown in the supplemental appendix.

16 It should be noted that the intervention variable is not binary (as it would be in a ‘project’) since insurance enrollment is measured as an average at the household level.

19