metadata: authors: 0: author: Alderman,Harold H.
1: author: Linnemayr,Sebastian
count: World,Africa
docna: 0: docna: Almost random : evaluating a large-scale
            randomized nutrition program in the presence of crossover
docty: Policy Research Working Paper
theme: Nutrition and food security
majtheme: Human development
subtopic: Health Care Services Industry,Nutrition,Energy and Mining
teratopic: Health, Nutrition and Population,Communities and Human Settlements,Finance and Financial Sector Development
abstracts: cdata!: Large-scale randomized interventions
            have the potential to uncover the causal effect of programs
            applying to a large population, thereby improving on the
            insights gained from currently dominant smaller randomized
            studies. However, the external validity gained through
            larger interventions typically implies less supervision and
            often comes at the cost of some deviation from the
            randomization plan. This paper investigates the impact of
            the Nutrition Enhancement Program, which aims to improve
            child nutrition in Senegal based on a large-scale randomized
            community intervention. The analysis explicitly deals with
            deviation from the planned treatment and suggests approaches
            for combining ex-post adjustments such as propensity score
            matching with the randomized treatment plan. The authors do
            not detect a strong overall program impact on the outcome
            measure of weight-for-age based on planned treatment status,
            but do find an impact on the youngest children. Moreover,
            the project impact is clearer when the analysis considers
            treatment crossover using alternative estimators of
            two-stage least-squares and propensity score matching. The
            findings underscore the importance of addressing the
            shortcomings of large-scale randomization interventions in a
            systematic manner in order to understand the selection
            process that can guide further implementation of such
            projects, as well as to expose the true, causal effect of
            such programs.
repnme: repnme: Almost random : evaluating a large-scale
            randomized nutrition program in the presence of crossover
display_title: Almost random : evaluating a
            large-scale randomized nutrition program in the presence of crossover
listing_relative_url: /research/2008/11/10087951/almost-random-evaluating-large-scale-randomized-nutrition-program-presence-crossover
topicv3: Nutrition m1326779 644,Iron m1503514 2931,Vitamin A m1504373 3634,Health m1504661 3480,Malnutrition m1627088 4737
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Health, Nutrition and Population,Communities and Human Settlements,Finance and Financial Sector Development
no_of_pages: 35
author_keys: 0: author_key: 95019
1: author_key: 630936
model_extractions: WPS4784 Policy ReseaRch WoRking PaPeR 4784 imPact evaluation seRies no. 28

# Almost Random

Evaluating a Large-Scale Randomized Nutrition Program in the Presence of Crossover _Sebastian Linnemayr Harold Alderman_ The World Bank Development Research Group Human Development and Public Services Team November 2008
Policy ReseaRch WoRking PaPeR 4784

# **Abstract**

Large-scale randomized interventions have the potential to uncover the causal effect of programs applying to a large population, thereby improving on the insights gained from currently dominant smaller randomized studies. However, the external validity gained through larger interventions typically implies less supervision and often comes at the cost of some deviation from the randomization plan. This paper investigates the impact of the Nutrition Enhancement Program, which aims to improve child nutrition in Senegal based on a large-scale randomized community intervention. The analysis explicitly deals with deviation from the planned treatment and suggests approaches for combining ex-post adjustments such as propensity score matching with the randomized treatment plan. The authors do not detect a strong overall program impact on the outcome measure of weight-for-age based on planned treatment status, but do find an impact on the youngest children. Moreover, the project impact is clearer when the analysis considers treatment crossover using alternative estimators of twostage least-squares and propensity score matching. The findings underscore the importance of addressing the shortcomings of large-scale randomization interventions in a systematic manner in order to understand the selection process that can guide further implementation of such projects, as well as to expose the true, causal effect of such programs.

This paper—a product of the Human Development and Public Services Team, Development Research Group—is part of a larger effort in the department to understand the impact of investments in health and nutrition. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at halderman@ worldbank.org.

_The Impact Evaluation Series has been established in recognition of the importance of impact evaluation studies for World Bank operations and for development in general. The series serves as a vehicle for the dissemination of findings of those studies. Papers in this series are part of the Bank’s Policy Research Working Paper Series. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Almost Random: Evaluating a Large-Scale Randomized Nutrition Program in the Presence of Crossover**

Sebastian Linnemayr<sup>♦</sup> Harvard University Harold Alderman<sup>‡</sup> The World Bank <u>Key words:</u> Nutrition; Impact evaluation <u>JEL Codes:</u> I12; O12

> ♦ Corresponding author: Sebastian Linnemayr, Harvard School of Public Health, Harvard University, Cambridge, USA; email: slinnema@hsph.harvard.edu.

> ‡ Harold Alderman, World Bank, Washington DC, USA; email: halderman@worldbank.org.
# **Acknowledgement of Data Organization and Collection; General Acknowledgement**

The data for this nutrition intervention program have been organized and collected by the Research Centre for Human Development under the direction of its director Salif Ndiaye, whom we owe gratitude. We also would like to thank Claudia Rokx as well as Biram Ndiaye, Coordinator of Cellule de Lutte contre la Malnutrition – Primature (PRN), Dakar during the period of this study, for collaboration on the design and implementation of the research. We would also like to thank the regional coordinators of the PRN and the regional organizers of the Forecasting and Statistics Directorate for their help. We gladly acknowledge the competence, enthusiasm, and availability of the members of the technical teams. The personnel on the ground, in particular the chauffeurs, interviewers, and team leaders have contributed significantly to the success of the survey. Last but not least we would like to thank the people who were interviewed for generously offering their time, energy, and commitment. We also thank Emaneula Galasso for comments on an earlier version.

1
# **1. Introduction**

Malnutrition is a persistent problem in developing countries. WHO estimates the fraction of malnourished children in developing countries at 33% as measured by the percentage of children stunted, i.e. children that fall below -2 standard deviations of the United States National Center for Health Statistics / WHO international reference median value (de Onis, Frongillo, and Blössner 2000). This has far-reaching consequences: malnutrition _in utero_ or in infancy can have a long-lasting negative impact on cognitive development and on their subsequent capacity to achieve sufficient income to provide for their own children when adults (Alderman, Hoddinott, and Kinsey 2006; Victora et al., 2008).

There is substantial agreement on the efficacy of a number of nutrition interventions (for a review see Allen and Gillespie 2001). There is little doubt, for example, that breastfeeding promotion saves infant lives, or that vitamin A prophylaxis reduces child mortality. However, there is little consensus that community based programs can reduce stunting or the risk of being underweight (Bhutta et al., 2008, Alderman, 2007).

This study assesses the impact of a package of health inputs on the anthropometric status of children in three regions in rural Senegal as evidenced by their weight-for-age. The relatively short time between the baseline and follow-up survey motivates the focus on two further measures of program success: health inputs and nutritional knowledge of the mothers. The program’s ability to influence these intermediary inputs not only helps explain the final outcomes but also indicates whether other health measures can be improved in the course of a program with a primary objective framed in terms of nutritional status. Furthermore, we stratify the sample by child age, i.e. whether the child 1
was exposed to the program in utero as well as during her lifetime. The short intervention duration makes it likely that the youngest children show the largest gain from the program that for longer durations could be expected to also raise the nutritional status of older children.

The current study makes two main contributions to the literature on evaluating randomized trials: first, we report the results of a large-scale nutrition intervention reaching 200,000 households. Second, the study illustrates the value of a randomized design even when there is departure from the initial design (Angrist et al., 2002). This is important since adherence to treatment status becomes increasingly difficult with the size of the project.<sup>2</sup> We illustrate that while the main policy result is robust to alternative approaches the empirical consequences of assumptions common in the literature are not trivial. Given the need for accurate evaluation of large-scale interventions for drawing lessons for scaling up the currently predominant small randomized studies, it is imperative to deal with deviations from initial treatment status and find ways to extract information from this situation.

The outline of the paper is as follows: in section 2, we describe the program. We then briefly summarize the identification strategies for the estimators in section 3 that we will use in the subsequent analysis. In section 4, we give an overview of the main variables used. Section 5 presents the results for nutritional status expressed as weight for age as well as looking at measures of behavioral change and the availability of health inputs based on the planned treatment status. Alternative estimators based on the actual receipt of the intervention are also presented. Section 6 concludes and discusses the differences in outcome between the approaches used to address imperfect randomization.

> 2 While close cooperation between the researcher and the implementation team on the ground reduces the deviation from the treatment assignment, it potentially also reduces the external validity of the study; such close supervision potentially also reduces the internal validity if the study objects feel closely observed.

2
# **2. Description of the intervention and its implementation**

# <u>2.1 Data setting</u>

The government of Senegal designed a strategy in 2002 to fight malnutrition that is scheduled to reach 50% of children under five years of age by the year 2011, the _Programme de Renforcement de la Nutrition_ – Nutrition Enhancement Program (PRN). The first phase of the Nutrition Enhancement Program targeted 20% of children under the age of five with growth promotion and the integrated service of child diseases at the community level. In the three regions under consideration (Fatick, Kaolack, and Kolda), child nutritional status is low and knowledge of best practices very limited: the percentage of children who are exclusively breastfed in the first four months of their lives varies between 1.1% in Kaolack and 2.6% in Kolda, even though the WHO recommends exclusive breastfeeding up to at least six months of age. Children in Senegal are also suffering from the lack of micronutrients that remains widespread despite interventions that have taken place in the past. 84% of children less than 5 years of age suffer from anemia, as do 61% of women. Vitamin A deficiency is a public health problem, with about 61% of children under the age of six years suffering from this deficiency.

Summary statistics for the sample population are presented in Table 1. The z-scores of weight-for-age is more than one standard deviation lower than the mean for the US reference group and indicates the poor nutritional standard of the children in the sample: for example, a two year old boy would on average weigh about 12.2 kg in the US; in Senegal, his average weight could be expected to be about 1.5 kg lower at about 10.8 kg (WHO Child Growth Standards).<sup>3</sup>

> 3 These standards can be accessed at http://www.who.int/childgrowth/en/.

3
Over the course of the first wave of the PRN, monthly growth promotion was provided to 200,000 mothers and their children with NGO agencies contracted to supply these services. One of the main pillars of the approach was monthly discussion with mothers related to nutrition organized at the community (i.e. village or villageneighborhood) level (Alderman et al., 2008). Great care was put into involving communities and key figures within these communities such as village elders, the marabou (a religious leader), or grandmothers, who traditionally play a big role in influencing feeding and child care practices. The goal of targeting these people in the implementation strategy is on the one hand to involve the agents actually influencing behavior of mothers, and on the other hand to educate these people who will then pass on the knowledge, thereby supporting the sustainability of the project. Meetings of pregnant women were organized in order to generate a forum where these women can exchange ideas and experiences concerning pregnancy and child-rearing. Another strategy encouraged is the principle of ‘positive deviation’: individuals who show behavior different form that of the other villagers and who avoid certain health problems are invited to share their experience and to teach the other women these novel strategies.

The following program components were carried out in all three regions: - Behavioral change towards exclusive breastfeeding without supplementation for at least the first six months of the child’s life was promoted in discussion groups

- Growth promotion: during these sessions, the health worker weighs the child and discusses its progress with the mother by comparing it to a growth chart distributed - Vitamin A supplementation: in the course of the weighing sessions, vitamin A is distributed to children 6 – 59 months and mothers in the 42 days after having given birth

4
- Iron supplementation: in discussion rounds, pregnant women are encouraged to take iron supplements that are distributed by health centers

- Bednets distribution for a fee (including a subsidy) and demonstration on their use

- Deworming was offered to all children aged 6 – 59 months.

- Cooking workshops were organized to demonstrate the preparation of nutritious

- foods for the mothers as well as supplements to breastfeeding after six months of age

# <u>2.2 Implementation of the randomized treatment status assignment</u>

As part of the first phase of the PRN, a randomized treatment assignment was implemented in three poor rural regions in order to allow for subsequent evaluation before scaling up to the rest of the country’s rural regions. The implementing NGOs were asked to provide a list of villages in which they had the means and intention to intervene. From the total list of about 1000 villages, 212 villages were randomly chosen in the three regions. Based on these villages, 220 clusters were identified (some villages are large enough to have 2 clusters, and one village had three clusters), and in each cluster up to 20 households were randomly drawn based on the list of households in the village. The nutrition intervention was randomly assigned to half of these villages; the NGOs were asked to schedule services to the other half in a later wave of implementation. They were free to include other villages not among the 212 in the intervention group at any time.

A baseline survey was conducted in April 2004 in all 212 villages, collecting data about the health status of the children, socioeconomic variables of the households these children are residing in, and extensive information about the nutrition and child care practices of the mother. The survey teams administered three questionnaires: a village 5
questionnaire, a household questionnaire, and an individual questionnaire for the mother of the child. If there was no child under three years of age in the household, the household was dropped in the first round without replacement. In the second round, these households were replaced with other randomly drawn households from the same village list until the fixed number of households was interviewed. This change in the survey design explains the significantly larger number of children measured in the second round. In June 2006 the same information as at baseline was collected in control and treatment villages.

# **3. Strategy for empirical analysis**

When evaluating the effectiveness of a treatment _T_ , we would like to compare the difference _D_ in the outcome variable of interest _Y_ for the same individual _i_ once he receives the treatment and once when he does not<sup>4</sup> :

> (1) _D_ = _YiT_ − _YiC_ ,

where the superscript _T_ denotes an individual receiving the treatment, and _C_ stands for the outcome without the treatment. As we cannot observe the same individual or unit in two states of the world at the same time we face the so-called problem of the missing counterfactual. However, it may be possible to discern the average effect of a certain intervention on a group of individuals:

> When subtracting and adding the unobserved but typically well-defined term _E_ [<sup>_Y_</sup> _iC | T_ ],

i.e. the outcome of the treatment group in the absence of treatment, we can state the

> 4  Key references on this topic include Duflo, Glennerster, and Kremer (2006) or Angrist and Krueger (1999).  For the original reference for the Rubin causal model see Rubin (1974).

6
evaluation problem as the situation in which the total change in the outcome consists of the treatment effect and the selection bias that confounds causal identification: Much of empirical work is concerned with finding ways to control for selection bias, the difference in the non-treatment outcome between treatment and control individuals. The challenge is to establish a close estimate of the missing observation of the non-treatment outcome of the treatment group.

Randomization guarantees on average for a large sample that in the absence of the intervention the control and treatment groups have the same outcome: (4) _E_ [ _YiC T_ ]− _E_ [ _YiC C_ ] = 0 , In such a situation, a simple comparison of the sample post-intervention means suffices to measure the average treatment effect of the intervention. In terms of regression analysis one can regress the outcome on covariates and a dummy variable for inclusion in the treatment group: (5) _Yit_ = _X it_ β+ _Tt_ δ + _eit_ , where _e_ is an error term composed of individual, family and community unobserved fixed characteristics as well as a stochastic disturbance term, μ it:

> (6)<sup>_e_</sup> _it_ = ν _i_ +η _i_ + ε _i_ + μ _it_ .

In small samples, there is the possibility that villages differ in their characteristics influencing outcomes. Therefore, it is common to include socioeconomic variables _X_ at the individual and household level that in previous studies have been shown to influence the outcome of interest (see for example Behrman and Skoufias 2004). Note that the introduction of control variables at the individual and household level should not change the estimate of _β_ unless _Z_ and _X_ are correlated (Angrist and Krueger 1999).

7
Random assignment is, however, not without its pitfalls. For example, individuals selected for the treatment may not take it up, so that the intention to treat does not provide an accurate assessment of the impact of the treatment on the treated. Or individuals assigned to the control group obtain the service from an alternative source, say a private provider. This is often called a crossover effect. Angrist et al. (2002) provide an illustration in which some individuals who received a randomly assigned school voucher did not utilize it and others in the control group received a functionally similar scholarship from private groups.

A variation of the cross over problem occurs when the implementation does not follow the program assignment strictly, as occurred in the current study. In such a situation, planned and actual treatment status differ, with the intent-to-treat estimator (as defined by the researcher) representing a lower bound on the evaluation estimate.

Another problem that can lead to a mitigated program impact is spillovers from treatment to control villages, for example, when information being disseminated during the intervention is shared. In such a situation, we would expect the outcome to improve for both groups, but more so for the treatment units given their longer and more intense exposure to the program.

One alternative to random assignment of treatments is to analyse impacts using a difference in difference method in the context of panel data. By construction, the fixed effects remove the corresponding fixed component of the error term and thus any correlation between it and the treatment variable _T_ . This simple problem can be implemented in a regression set-up with two data-waves such as: (7) _Yit_ = α + β⋅ (1 _i_ ∈ _T_ ) +γ ⋅ (1 _t_ = 2) + δ⋅ (1 _t_ = 2) *(1 _i_ ∈ _T_ ) + ε _it_ , 8
where the second term on the right hand side controls for initial differences between the control and the treatment group, the third term controls for a time-trend common to both groups, and the fourth term indicates the treatment effect of actually receiving the intervention. The estimate of interest is _(δ-β)_ which measures the treatment effect purged of initial differences under the assumption that in the absence of treatment both groups would experience a similar trend in the outcome variable. When there are deviation from planned treatment it is necessary to assume that these are not correlated with responsiveness to the program since such responsiveness being a matter of heterogeneity of response is not removed by differencing.

Moreover, in cases in which the assignment is based on observed values of the outcome desired – say, where the treatment is prioritized to groups with low test scores or nutritional status –it is likely that [ _E_ ( μ _it_ | _T_ ) ≠ 0] since measurement error partially determines the assignment. This might be the case in our sample; as discussed further below, villages with lower initial nutritional status had a higher probability to be included in the treatment group. If this reflects fixed effects, then difference in difference will address the bias, but if it reflects time varying factors, including measurement errors, then difference in difference will not solve the bias of reversion to the mean. Chay et al. (2005) present an example of an assignment to treatment based on baseline performance where difference-in-differences results are biased.

For the analysis at hand, the treatment group includes about 30% of households that did not receive the intervention that was planned; thus, the intent-to-treat estimator can be viewed as a lower bound for the magnitude of the nutrition intervention. However, the potential bias can be addressed by using the planned treatment status instrument for receipt of the actual intervention as in Angrist et al. (2002). Planned 9
treatment status is exogenous by construction yet being a strong predictor of actual treatment fulfills the requirements for instrumental variables. In the first stage, planned treatment status and other village-level variables _Z_ are used as instruments for actual receipt of the intervention: In the second stage, the fitted value of _T_ is used in the regression: Yet another way of addressing imperfect execution of random assignment is to use propensity score matching (Rosenbaum and Rubin, 1983). If the potential treatment and control outcomes are independent when conditioning on some observable characteristics, then the outcomes are also independent conditional on the propensity score. Smith and Todd (2005) review the performance of matching estimators and come to the conclusion that for their sample a difference-in-difference matching estimator performs best, motivating the use of this estimator in the current study. Hirano et al.

(2003) suggest weighting the observations by a function of the estimated propensity score to arrive at a more efficient estimator. Their approach can be implemented by using the ^ ^ weight of 1/ _P_ for villages having actually received the treatment, and 1/(1 − _P_ ) for both control villages and treatment villages not having received the intervention.

The planned treatment can be used in estimating the weights. This combination of approaches also addresses a problem that is discussed in Heckman et al. (1998), that of common support in matching approaches. When the major explanation for the selection into treatment remains the planned treatment status which was random, there is ample overlap between the treatment and control villages. Additionally, the combination of 10
propensity score estimates with difference in difference helps address the possibility that the assumption of time independent selection does not hold (Ravallion, 2008).

Several studies have used both approaches in order to evaluate the degree of selection bias to which observational studies may be subjected. For example, Lalonde (1986) finds that there are often significant differences between prospective and retrospective empirical approaches for the evaluation of programs as does Glewwe et al. (2004).<sup>5</sup> The current paper aims to contribute to this discussion by employing alternative approaches using the same dataset.

# **4. Empirical Analysis**

# <u>4.1. Measures of program success used</u>

The main focus of the evaluation of the current nutrition intervention is to assess the impact of the set of services offered on the nutritional status of children. When preparing the intervention, the outcome measure of weight-for-age z-score was determined as the original indicator to determine the sample size of the program as well as the indicator for tracking success of the program on the part of the organizations financing the PRN<sup>6</sup> .

The outcome of interest reflects a package of services which are valued not only for their impact on weight but also as indicators of the functioning of a community health program in general. For this reason, we also examine the availability of health care measures before and after birth such as micronutrient supplementation or malaria bednets. The analysis of these measures can provide supportive evidence of program

> 5 For overviews see Duflo, Glennerster, and Kremer (2006) and Ravallion (2008).

> 6 Apart from weight-for-age, we also investigated the impact of the program on height-for-age, another frequently used anthropometric measure that captures more long-term impacts. The results obtained for this measure are consistent with the ones for weight-for-age and are therefore not presented to reduce the number of output tables.  However, they have fewer observations since recumbent length – notoriously difficult to measure – was not collected on younger children.  The omitted tables are available from the authors.

11
success (in the sense that the health inputs reached the villages/households) that are less prone to measurement error, rely on faster-moving measures, and indicate potential future success of the program if it takes time to transform these input measures into measurable change in the outcome z-scores.

The unit of assignment in the current study is the village and not the individual; all households in a village belong either to the control or the treatment group. As a result, actual take-up of the program by individuals is not observed, although the village health workers tried to encourage all mothers in the village to participate in the program. Therefore, it is the impact of the availability of the program in a village rather than its actual take-up that is evaluated. As actual take-up is a choice variable, investigating the effects of availability of the program may be less prone to selection bias (Strauss 1990).

In addition to investigating health inputs as well as health outcomes, we also try to take potential heterogeneity of the program impact into account. In particular, we test whether younger children have a different response than the older cohort. Given that the program was available for less than two years in the villages, we test the possibility that the children that were exposed to the intervention already _in utero_ have greater benefits. This would be in keeping with evidence that malnutrition occurs at very early ages and is fairly unresponsive by 18 months (Shrimpton et al. 2001). The effectiveness of the program may also depend on village initial infrastructure, a possibility we take into account in the analysis below by stratifying by whether the village has a road to the next village that can be used all year round, and also along the average village wealth at baseline.

<u>4.2. Control of the success of the randomization of planned treatment status</u> 12
In the next step we investigate whether the assignment of treatment status to the villages happened in a random fashion, in which case the outcome variables as well as the conditioning variables should not differ between the treatment and the control group in the baseline period except for random deviations (Behrman and Todd 1999). The comparison of villages along their planned treatment status in Table 2 shows that the two types of villages have generally similar outcome and control variables.

As discussed above, while the initial assignment of treatment was random, the NGOs implementing the program may not have fully understood the importance of the randomization procedure prescribed, or deviated from the planned treatment status of villages for other reasons. In Table 3, we see that there was, in fact, deviation of actual treatment status from the status initially assigned. Of the 111 villages that were initially assigned treatment status, 80 ended up receiving treatment, while 31 villages (28%) did not receive the intervention. Of the 100 initial control villages, for one village no second round data were available, and of the 99 remaining villages, 8% received the intervention despite their control status. In the analysis to follow, we therefore have to address the problem of partial compliance.

When we compare the most important group of villages that deviated from the original design, the 31 villages that initially were assigned treatment status but that did subsequently not receive intervention, along some key dimensions with the remaining 80 villages from the group initially assigned treatment status there is some evidence that the deviation did not happen randomly (Table 4). The villages from the 2004 planned treatment list that subsequently did not receive the nutrition intervention were initially somewhat better off in terms of their nutritional status and had less children that were mildly underweight (below -2 standard deviations from the mean of the US reference 13
population). They had a market more often and showed a lower presence of NGOs and health posts than the villages that retained their treatment status. These statistics indicate the possibility that the NGOs purposely selected villages by focusing on the worst-off villages in the treatment sample and on those villages in which there was already a NGO present (possibly the intervening NGO itself).

Basing the analysis on actual rather than planned treatment-status could lead to a systematic underestimation of the treatment status if, by targeting the worst-off villages, the service providers gave priority to villages that were less able to profit to the same degree from the nutrition intervention program compared to their better-off peers. On the other hand, if the selection was based on short term fluctuations in nutritional status, we might see an improvement in these villages that was partly due to a reversion to the mean as discussed in section 3. This would tend to overestimate the positive effect of the intervention. While selection on fixed community characteristics can be addressed using difference-in-differences analysis that is employed, as mentioned, this step would not necessarily be the case if the selection criteria included time varying factors. A similar caveat applies for the propensity score estimator that makes the assumption that the selection bias is time-invariant (and can hence be removed in a panel context) conditional on observables. Below, we will contrast the outcomes of the analyses based on planned and actual treatment, using both the intent-to-treat and a 2SLS estimator using planned treatment status as an instrument; for the analysis based on actual treatment status we focus on treatment-on-the-treated as well as a matching estimator combined with difference-in-differences that compares villages with similar probability of having been selected into treatment.

14
# **5. Results**

The first columns in Table 5 present the results for weight-for-age using the original randomization classification in 2004, irrespective of whether the villages actually received the program. The approach uses both the baseline and subsequent data and, thus, combines difference in difference with random assignment. The coefficient for the variable _planned intervention_ indicates the difference of the mean value for the villages that were initially assigned treatment status from those expected to be in the control group. We include age dummies of the children in six months age groups, with the children between 30 months and 3 years of age representing the omitted group. The original treatment status assigned to the village irrespective of actual receipt of the program is an imperfect indicator of services actually delivered. As such, using the planned assignment status avoids a correlation with unobserved factors at the possible expense of increasing errors in variable from mismeasurement from which an attenuation bias is expected.

Based on the two assumptions that in the absence of treatment, the villages in both groups would experience a similar trend in malnutrition rates and also that selection was not based on the time varying factors such as the level of the outcome at time the baseline was implemented a difference in difference framework using the actual receipt of treatment would give an accurate assessment of the impact of the services. This approach, however, could introduce a bias and, unlike the bias due to attenuation with the planned treatment, the direction of such a possible bias is unknown. We report the outcome based on the assumption of a time-invariant selection bias in column 3. In all specifications, we allow for the clustering of standard errors at the village level.

15
For the specification based on planned treatment status in column (1), we do not find a statistically significant impact in the program villages in comparison to the control villages. In contrast, when basing the analysis on actual treatment status, the program is found to increase weight-for-age of children 0-36 months by one-tenth of a standard deviation on average. The age dummies (omitted for space reasons) reflect the common finding of a deterioration of the nutritional situation for the children with increasing age when compared to the children in the reference group. The other control variables mirror the findings in previous studies on the determinants of child nutrition: parent’s education and sanitary facilities in the household improve the nutritional status, while the status of being a twin reduces it significantly. The gender dummy is insignificant, as discrimination by gender is typically not observed in Africa (Svedberg, 1990). As indicated by the indicator variable ‘ _second round’,_ the villages in the sample experience an overall increase in the weight-for-age indicator of about one tenth of a standard deviation for both types of villages that is statistically significant at the 1% level. This finding confirms the general trend observed in the summary statistics in Table 1.

There are several reasons why the program impact may differ by the age at which the children were exposed to treatment (Alderman, 2007). Children who at the time of the baseline survey in April 2004 were six months old were included in the second wave in 2006 although these children likely were weaned by the time the intervention began. In contrast, a child born after April 2004 would have had the additional benefit of their mothers participating in the discussion groups and micronutrient provision for pregnant women, two important program components. There is increasing evidence that the experiences _in utero_ can have long-lasting effects (Behrman and Rosenzweig 2004; 16
Strauss 2000). Additionally, these children would benefit from the advice to the mothers on the use of colostrum as well as on exclusive breastfeeding.

We therefore create a dummy variable termed ‘full exposure’ for children up to 6 months of age as these children have experienced any benefits the program provides during their entire life as well as _in utero_ . The results are presented in columns 2 (planned treatment status) and 4 (actual receipt of the intervention) of Table 5. It appears that the youngest children, i.e. those whose mothers benefited from the program when they were pregnant, benefitted from the intervention even though the average impact on children less than three years of age was virtually zero.<sup>7</sup> However, as discussed, the results in Table 5 are likely an underestimate of the true impact is based on planned treatment yet also possibly subject to reversion to the mean if based on actual treatment status. Thus, we instrument actual receipt of the treatment with planned treatment status, a variable that fulfills both criteria for a good instrument: it is both of exogenous character (by construction), and it is highly correlated with the actual receipt of the intervention. In addition, to the planned treatment, we add initial village-level characteristics such as distance to the next village, prevalence of female education, or the presence of a market that may have influenced the NGOs’ placement decision as well as five interaction of village characteristics and the planned treatment. Each of these interactions as well as the treatment status itself are individually statistically significant at p<.01. The r-square of the instrumenting equation is 0.72.

The instrumented treatment status in Table 6 does not have a statistically significant impact on child nutritional status using the full sample, although the sign of the coefficient has changed from negative as in Table 5 column 1 to positive. In column 2

> 7 As full exposure correlates with the age least susceptible to malnutrition, the dummy variable is positive as expected.  This however, does not affect the interaction term which measures the difference in difference.

17
we report the results with an age specific interaction and find again that the youngest children benefit substantially from the program. As the variables used for instrumentation of actual receipt of the treatment do not show variation at the individual level this regression is not run as a simultaneous system. Instead, the actual receipt of the intervention was predicted using the instruments discussed above. This variable was then interacted with the age of the child and introduced in the regression. The standard errors were then calculated by a bootstrap method using 100 repetitions for the second stage regression.

When stratifying according to initial conditions, we observe a positive and statistically significant impact in villages that are deprived of a road connecting them to the next village that is useable all year round. In villages connected with such a road, there is no discernible effect of the program when using planned treatment status as an instrument in the first stage regression. A similar, albeit only borderline statistically significant coefficient (p-value: .11) is observed for villages below the mean of the wealth index created from the possession of physical assets and livestock. For villages over that cut-off, instrumented planned treatment status is not statistically different from zero.

To explore the program impact further we look at additional measures of health care choices since the outcome variable weight-for-age is a function of behavioral inputs as well as health and time inputs. These latter indicators may change in response to the intervention relatively quickly and potentially translate into nutritional improvements with a delay. Health inputs such as drugs or bednets distributed also indicate whether the NGOs actually delivered the services required to the villages. The results for these inputs into the health production function are presented in Table 7, which is, again, an intention18
to-treat estimator based on the planned treatment status.<sup>8</sup> That is, we see if changes in these inputs are associated with the program. We can not, however, ascertain, which if any changes account for the overall impact.

The results in rows 1 and 2 indicate that the wide-spread practices of giving liquids other than breastmilk in the child’s first six months as well as the practice of not giving the colostrum after birth are less prevalent in planned intervention villages than in initial control villages in 2006, despite there not having been a significant difference in the prevalence between the groups in 2004. When expressing the coefficient in terms of marginal probabilities, we find that there was a reduction in the probability of giving liquids other than breast milk in the first three days following birth of 11% in the treatment group as compared to the control group.

For health inputs, two out of three measures we observe a statistically significant impact of planned treatment status. For the provision of bednets, the coefficient is not significant at the 10% level. Similar results are found for the provision of micronutrients: for vitamin A for infants and iron supplements for pregnant mothers, there is a statistically significant impact of being in a planned treatment village for vitamin A, and a borderline significant effect for iron supplementation. The last two rows in Table 7 show that disease prevalence is not affected by planned treatment status. This finding is not surprising as disease prevalence is likely to be correlated with the outcome measure of weight-for-age, for which we also found no significant impact between the planned treatment and control villages for the pooled sample.

The results for behavioral indicators and health inputs based on actual instead of planned treatment status confirm the findings in Table 7 and are omitted for space reasons as are results using an instrumented treatment. The above results indicate that the

> 8 The variable definitions for Table 7 are given in the appendix table A.1.

19
NGOs delivered health inputs and improved the knowledge of best practices in the planned treatment villages. Unfortunately, these behavioral and input changes did not translate into changes in the prevalence of diseases such as diarrhea that count among the main reasons for the low nutritional status of children in Senegal.

As discussed above, the planned treatment status can not only be used as an instrument but can also improve the overlap of the support for the propensity score for villages receiving the treatment and those that do not benefit from the intervention in conjunction with difference in difference estimates. Table 8 reports such an analysis following Hirano et al. (2003) and can be interpreted as the impact of actual receipt of the intervention for villages similar in their propensity score.<sup>9</sup> The results indicate coefficients that are statistically significant at the 5% level and sizeable in magnitude, with about a .27 standard deviation increase in weight-for-age score. Somewhat surprisingly, we do not find that younger children benefit relatively more from the intervention than their older peers. However, given that we find an average positive effect on weight-for-age for all age groups, the extra effect for young children may be incorporated in the average impact in the villages.

# **6. Conclusion**

The aim of the current study is to evaluate the success of a pilot program forming part of the _Programme de Renforcement de la Nutrition_ , a nutrition intervention program targeted at young children in Senegal that introduces the program components to three poor rural regions. Identification of the treatment effects is based on the random assignment of the treatment status among 212 villages in April 2004 before receiving the

> 9 Following common procedure in the literature, we constrict our sample to observations with a propensity score lying in the interval [.05; .95].

20
intervention and being re-surveyed in June 2006. However, given substantial deviation from the assigned treatment status, we compare these results with approaches based on the actual receipt of the intervention. The planned treatment status is used both as an instrument that is plausibly exogenous by construction, but can also be used as an input into the propensity score in a matching approach.

We find significant changes in health care practices in the villages assigned to the treatment status. But using this assignment as an indicator of treatment, we do not find an average overall impact on weight-for-age of children. We do, however, observe that those children whose mothers benefit from the intervention during their pregnancy display a significantly improved nutritional status than their older peers who were likely weaned before the program began. These observations can guide the allocation of resources in similar programs.

However, while these core results give an indication of the project’s success, the magnitude of the impact is biased downwards due to cross over effects. Thus, we also report results using differences in differences based on the actual treatment status as of June 2006 instead of the planned one from 2004 as well as an instrumental variables approach based on the planned treatment assignment. Both approaches to evaluating the study tend to results in larger estimates of the treatment effects compared to the results based on planned treatment status alone. These results shed additional light on the impact of a large-scale nutrition intervention in a situation where adherence to the assigned treatment status is less than perfect.

21