metadata: authors: 0: author: Cameron, Lisa
1: author: Olivia,Susan
2: author: Shah, Manisha
count: Indonesia
docna: 0: docna: Impact evaluation of a large-scale rural
            sanitation project in Indonesia
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Sanitation and Sewerage,Engineering,Gender and Development,Water and Human Health,Hydrology,Water Supply and Sanitation Economics,Sanitary Environmental Engineering,Small Private Water Supply Providers,Environmental Engineering,Health and Sanitation,Town Water Supply and Sanitation
teratopic: Water Supply and Sanitation,Health, Nutrition and Population,Communities and Human Settlements
abstracts: cdata!: Lack of sanitation and poor hygiene
            behavior cause a tremendous disease burden among the poor.
            This paper evaluates the impact of the Total Sanitation and
            Sanitation Marketing project in Indonesia, where about 11
            percent of children have diarrhea in any two-week period and
            more than 33,000 children die each year from diarrhea. The
            evaluation utilizes a randomized controlled trial but is
            unusual in that the program was evaluated when implemented
            at scale across the province of rural East Java in a way
            that was designed to strengthen the enabling environment and
            so be sustainable. One hundred and sixty communities across
            eight rural districts participated, and approximately 2,100
            households were interviewed before and after the
            intervention. The authors found that the project increased
            toilet construction by approximately 3 percentage points (a
            31 percent increase in the rate of toilet construction). The
            changes were primarily among non-poor households that did
            not have access to sanitation at baseline. Open defecation
            among these households decreased by 6 percentage points (or
            17 percent). Diarrhea prevalence was 30 percent lower in
            treatment communities than in control communities at endline
            (3.3 versus 4.6 percent). The analysis cannot rule out that
            the differences in drinking water and handwashing behavior
            drove the decline in diarrhea. Reductions in parasitic
            infestations and improvements in height and weight were
            found for the non-poor sample with no sanitation at baseline.
geo_reg_key: 517198,517191
repnme: repnme: Impact evaluation of a large-scale rural
            sanitation project in Indonesia
display_title: Impact evaluation of a large-scale
            rural sanitation project in Indonesia
listing_relative_url: /research/2013/02/17288401/impact-evaluation-large-scale-rural-sanitation-project-indonesia
topicv3: Sanitation m1326453 2000,Water m1326522 137,Gender m1327621 615,Health m1504661 3480,Latrines m1505177 3454
publishtoextweb_dt: 2013-02-02T00:00:00Z
historic_topic: Water Supply and Sanitation,Health, Nutrition and Population,Communities and Human Settlements
no_of_pages: 59
author_keys: 0: author_key: 1174522
model_extractions: WPS6360 Policy Research Working Paper 6360 Impact Evaluation Series No. 83 Impact Evaluation of a Large-Scale Rural Sanitation Project in Indonesia _Lisa Cameron Manisha Shah Susan Olivia_ The World Bank Sustainable Development Network Water and Sanitation Program February 2013
Policy Research Working Paper 6360

# **Abstract**

Lack of sanitation and poor hygiene behavior cause a tremendous disease burden among the poor. This paper evaluates the impact of the Total Sanitation and Sanitation Marketing project in Indonesia, where about 11 percent of children have diarrhea in any two-week period and more than 33,000 children die each year from diarrhea. The evaluation utilizes a randomized controlled trial but is unusual in that the program was evaluated when implemented at scale across the province of rural East Java in a way that was designed to strengthen the enabling environment and so be sustainable. One hundred and sixty communities across eight rural districts participated, and approximately 2,100 households were interviewed before and after the intervention.

The authors found that the project increased toilet construction by approximately 3 percentage points (a 31 percent increase in the rate of toilet construction). The changes were primarily among non-poor households that did not have access to sanitation at baseline. Open defecation among these households decreased by 6 percentage points (or 17 percent). Diarrhea prevalence was 30 percent lower in treatment communities than in control communities at endline (3.3 versus 4.6 percent). The analysis cannot rule out that the differences in drinking water and handwashing behavior drove the decline in diarrhea. Reductions in parasitic infestations and improvements in height and weight were found for the non-poor sample with no sanitation at baseline.

This paper is a product of the Water and Sanitation Program (WPS), Sustainable Development Network. WSP is a multidonor partnership created in 1978 and administered by the World Bank to support poor people in obtaining affordable, safe, and sustainable access to water and sanitation services. The paper is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at lisa. cameron@monash.edu.

_The Impact Evaluation Series has been established in recognition of the importance of impact evaluation studies for World Bank operations and for development in general. The series serves as a vehicle for the dissemination of findings of those studies. Papers in this series are part of the Bank’s Policy Research Working Paper Series. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Impact Evaluation of a Large-Scale Rural Sanitation Project in Indonesia**<sup>**1**</sup>

Lisa Cameron Manisha Shah Susan Olivia

**JEL codes** : I15, J10, O12 **Keywords** : rural sanitation, child health

1 Lisa Cameron is a professor in the Department of Econometrics at Monash University and director of the Monash Centre of Development Economics (MCDE). Manisha Shah is an assistant professor of public policy at the University of California, Los Angeles, and a faculty research fellow at the National Bureau of Economic Research (NBER). Susan Olivia is a postdoctoral fellow in the Department of Econometrics at Monash University. The project’s global impact evaluation task team is led by Bertha Briceño (in its early stages, it was led by Jack Molyneaux), together with Alexandra Orsola-Vidal and Claire Chase. Professor Paul Gertler has provided guidance and advice throughout the project. Advisors also include Sebastian Galiani, Jack Colford, Ben Arnold, Pavani Ram, Lia Fernald, and Patricia Kariger.The authors are also grateful to Almud Weitz, the regional team leader for WSPEast Asia, and Eduardo Perez, the global task team leader for the Scaling Up Rural Sanitation program. The WSPIndonesia country technical assistance team was led by Nilanjana Mukherjee, followed by the late Ratna Indrawati Josodipoero, and is now headed by Deviariandy Setiawan. Generous financial support was provided by the Bill & Melinda Gates Foundation.
# **1. INTRODUCTION**

Of the four most important causes of mortality for children under five years old in Indonesia, two (diarrhea and typhoid) are fecal-borne illnesses directly linked to inadequate water supply, sanitation, and hygiene issues (Ministry of Health, 2002). In Indonesia, about 11 percent of children have diarrhea in any two-week period and it has been estimated that more than 33,000 children die each year from diarrhea and 11,000 from typhoid (Curtis, 2004). Inadequate sanitation is associated not only with adverse health effects, but also with significant economic losses. A recent estimate of the economic losses in Indonesia from inadequate sanitation and poor hygiene practices puts them at more than 2.4 percent of the country’s gross domestic product (GDP), or approximately US$6.3 billion (Napitupulu and Hutton, 2008).<sup>2</sup> Despite wide recognition of the consequent health burdens, a large fraction of the world’s population still lacks access to basic sanitation. For instance, it is estimated that 110 million people in Indonesia lack access to proper sanitation (WHO/UNICEF, 2012). Of these, 63 million practice open defecation (WHO/UNICEF, 2012). Open defecation is of fundamental importance to development because of the health hazard it poses to anyone living nearby. If some members of a community continue to defecate in the open, the whole community is at greater risk of diarrheal diseases than people living in communities where open defecation is not practiced. Preventable diseases that result from lack of sanitation and poor hygiene behavior cause a tremendous disease burden among the poor. The greatest disease burden falls on infants and children under five years old (Murray and Lopez, 1997). By reducing normal food consumption and nutrient adsorption, diarrheal diseases are also a significant cause of malnutrition, leading to impaired physical growth and cognitive development (Guerrant et al., 1999), reduced resistance to infection (Baqui et al., 1993), and, potentially, long-term gastrointestinal disorders (Schneider et al., 1978). Improvements in sanitation thus can help reduce the transmission of pathogens that cause diarrhea by preventing human fecal matter from contaminating environments.

In January 2007, the Water and Sanitation Program (WSP) launched the Scaling Up Rural Sanitation program<sup>3</sup> to address the precarious sanitation conditions of large rural populations in the developing world. With technical support from WSP, local and national governments are implementing the program in three countries: Indonesia, India, and Tanzania.

The Scaling Up Rural Sanitation program focuses on learning how to combine the approaches of Community-Led Total Sanitation (CLTS), behavior change communications, and social marketing of sanitation to generate sanitation demand and strengthen the supply of sanitation products and services at scale, leading to improved health for people in rural areas. The project

> 2 A recent four-county study estimate found that the annual economic impact of poor sanitation is approximately US$1.4 billion (1.5 percent of GDP) in the Philippines, US$780 million (1.3 percent of GDP) in Vietnam, and US$450 million (7.2 percent of GDP) in Cambodia (WSP, 2008).

> 3 For more information on the Scaling Up Rural Sanitation program, see www.wsp.org/scalingupsanitation.

2
also seeks to support government efforts to develop strong enabling environments to reform policy and sector institutions to create large-scale sustainable programs.

To test the effectiveness of its approach, the Scaling Up Rural Sanitation program incorporated a randomized controlled trial impact evaluation (IE) in the three project countries as a key component of its monitoring and evaluation framework. This evaluation provides a unique opportunity to learn what health and welfare impacts can be expected from sanitation improvements. The IE uses widely accepted impact evaluation protocols, and was designed to disrupt the planned program as little as possible. Importantly, the IE evaluates an intervention that was implemented at scale and led by the government under real-world conditions. This unusual approach is in contrast to smaller randomized controlled trials that are carried out with the intervention as a pilot and under much more controlled circumstances. By evaluating the project at scale, the IE provides a more reliable estimate of program impact in a scaled up, sustainable program.

The evaluation seeks to examine the program impacts along the causal chain. Poor sanitation leads to diarrhea and lack of nutrients. This adversely affects child height and weight and can lead to anemia. Anemia in young children is associated with poor child development. Thus, key outcome variables of interest are:

- changes in perceptions of consequences of poor sanitation;

- sanitation improvements (toilet construction and access to improved sanitation);

- place of defecation (reduction in open defecation);

- child health outcomes, which include:

- diarrhea prevalence,

- intestinal parasite infections,

- stunting and wasting,

- iron-deficiency anemia detected through minimally invasive finger-prick tests, and

- cognitive and motor development.

This report aims to provide endline information for the selected indicators and outcomes of interest included in the study.

# **2. BACKGROUND AND DESCRIPTION OF THE PROGRAM**

In Indonesia, the rural sanitation project is known as Total Sanitation and Sanitation Marketing (TSSM). TSSM aims to improve sanitation practices in rural communities of East Java by generating sanitation demand at scale and increasing the supply of sanitation products and services. The project’s approach differs from the government’s previous established sanitation 3
policies of providing infrastructure and/or subsidies, instead sending facilitators to villages to initiate participatory analysis of existing sanitation practices and the consequences and implications of such practices, thus generating demand for better sanitation services that the market can then respond to.

In Indonesia, the project’s programmatic approach consists of three main components:

# _Community-Led Total Sanitation (CLTS)_

This component focuses on stopping open defecation. It aims to trigger the desire for an opendefecation-free (ODF) community. It does this by raising collective awareness of the open defecation problem. Facilitators are sent to communities to initiate analysis and discussions of the sanitation situation. These discussions are held in public places and are open to all. They involve a “walk of shame,” during which the facilitator helps people analyze how fecal contamination spreads from the exposed excreta to their living environments and food and drinking water. A map of the village is drawn on the ground and villagers are asked to indicate where they live, where they defecate, and the routes they take there and back. It soon becomes apparent that everyone is ingesting small amounts of each other’s feces (to people’s horror and embarrassment). This inevitably leads to personal and collective decisions to be free of the hazard by becoming an ODF community. The villagers must forge their own plan for making this happen with only limited follow-up support and monitoring from the program. ODF status is verified by local government agencies. Communities achieving ODF status receive recognition and commendation from local and provincial governments.

# _Social Marketing of Sanitation_

This component focuses on increasing the availability of sanitation-related products (such as latrines or slabs) and services (such as pit emptying). It involves extensive consumer and market research that investigates the sanitation solutions that people desire, the options available to them in the market, and their attitudes and knowledge of sanitation issues. The component develops targeted communications campaigns and enhances the supply of a range of sanitation goods and services that are responsive to preferences and economic capacities of all consumer segments. It also involves partnering with the local private sector—for example, to train local artisans to meet the increased demand for specific products that results from CLTS facilitation sessions.

# _Strengthening the Enabling Environment_

This component aims to support the development of policies and institutional practices that facilitate scaling up, program effectiveness, and sustainability. These include national, state, and local government sanitation policies; sanitation program financing, implementation, and management practices; fiscal rewards for results consistent with policies; training and 4
accreditation of facilitators, masons, and vendors; and regulation and support of local private sector investment in improving sanitation.

# **3. EVALUATION DESIGN**

Accurately measuring the long-term health and welfare impacts of these sanitation interventions requires an IE methodology that establishes the causal linkages between the intervention and the outcomes of interest. To estimate the causal relationship between the project (treatment) and the outcomes of interest, one needs an accurate counterfactual—that is, a comparison group that shows what would have happened to the target group in the absence of the intervention. The IE methodology uses randomization to construct the comparison group. Some communities were randomly selected to receive the treatment while others were randomly selected to serve as controls and not to receive the treatment within the period of the evaluation. If a nonrandom control group was used instead, a comparison of treated and untreated areas could confuse the program impact with pre-existing differences between villages. This is a particular problem if communities are chosen purposely as areas with a high likelihood of success because of favorable local conditions (strong leadership, existing water and sanitation infrastructure, highly educated populations, etc.) or because they differ in terms of hygiene habits, lower motivation, or other factors that are difficult to observe. This is known as selection bias. The use of randomization avoids these difficulties by ensuring that the communities that receive the intervention are no different on average from those that do not.

TSSM in Indonesia was implemented in rural East Java. Eight of 29 rural districts in East Java participated in the evaluation, with a total of 160 communities participating. The map in Figure 1 shows the location of the eight IE districts, which are spread across East Java. They are Probolinggo, Bondowoso, Situbondo, and Banyuwangi in the east of the province, and Ngawi, Madiun, Jombang, and Blitar in the west of the province.

5
<!-- Start of picture text --> Impact Evaluation - Total Sanitation<br>and Sanitation Marketing (TSSM)<br>Ngawi<br>280000 + + Kd + + + + + 280000<br>9150000: é 4 “G A caet 9180000<br>ttre<br>— HV ae ani<br>8100000" mee= + & a 2 #100000<br>ee fi 6 4° vo Ro~| s \ i<br>_<br>Blitar AJ oo oes<br>soso + + + + + + + oo pies soto<br>Legend 20 i) 20 40 Kilometers j ,<br><!-- End of picture text -->
In each of the participating districts, the IE team randomly selected 10 pairs of villages. Each pair consisted of one treatment village and one comparison village from the same _kecamatan_ (sub-district). Villages in Indonesia have various communities or sub-villages, and the project intervention occurred at the sub-village level. At least one community in the treatment village received the full project intervention. No communities in the comparison villages received the project intervention.

# **4. SAMPLING STRATEGY**

# _4.1 Selecting Sub-villages_

From each of the eight districts, 10 treatment and 10 control villages were randomly chosen to participate in the IE. Local government offices from each district gave the IE team a list of at least 30 villages in which the program could be implemented.<sup>5</sup> Most district offices gave the IE teams lists of 40 to 70 villages. These are villages the districts had chosen to participate in the project based on sanitation needs, poverty levels, access to water, and so forth. Using a random number generator in STATA, the IE team randomly selected 10 treatment and 10 control villages from each district list. The IE team then sent the list of 20 villages back to the district government office (without telling them which villages had been selected as control and treatment villages). The reason for this is that the project is actually implemented at the _dusun_ , or sub-village level. Villages generally have two to three sub-villages. Wanting the same selection criteria to be used for the selection of sub-village for both the treatment and control villages, the IE team asked each district office to provide the sub-village names for all 20 villages. District offices were told that some would be the treatment and others the control. The IE has internal validity but not external validity in that villages were not randomly chosen from the universe of villages. Different districts chose villages on the basis of different indicators. For example, some districts chose to include villages that had recently participated in water supply programs, whereas other districts explicitly chose to exclude such villages. The sample thus reflects the variety of ways in which government officials generally choose villages for a sanitation program so internal validity is sufficient under these circumstances. That is, the evaluation will provide estimates of the average impact expected given the way governments select villages for such programs. The impact of the different bases for the choices can be examined as part of the evaluation.

Once the IE team received the sub-village lists from the district offices for all 20 villages, they told district offices which villages were in the treatment group and which were in the control group. The district offices promised to do everything possible to make sure the treatment _dusun_ were treated and the control _dusun_ remained untreated. There was some concern by local

> 5 At least two villages were listed for every sub-district on the district lists. This allowed pairing of treatment and control communities with sub-districts as described above.

7
program implementers that the program might spread like “wildfire” and that it would be difficult to deny it to control villages. However, sample sizes were selected based on this possibility and it does not appear that many control villages were contaminated.

# _4.2 Selecting Households_

Listings were done in each sub-village in the control and treatment villages to gather information on the universe of households with children under the age of two years. These listings were based on information provided by the community health cadre. Thirteen households were then randomly selected from the listing to participate in the baseline survey. These 13 households were given priority rankings so survey teams knew to interview them. If one of the 13 households was unavailable to participate, it was replaced by another household chosen randomly from the listing. Detailed replacement methods are described below. In some of the sub-villages, there were not enough households with children under the age of two years. In those cases, information on households with children under age five was also collected. These households were ranked with priority rankings based on the total number of child under the age of two years, under the age of three years, under the age of four years, and under the age of five years. Households with younger children were given a higher priority.

Households in the sample had at least one living child under the age of two (unless there were not enough households with children under age two in the sub-village). If the child under age two had died or moved since the listing was conducted, the decision-making process was as follows:

1. Are all listed children under age two in this household deceased? If yes, is there another child under age two in this household? If yes, conduct interview. If no, replace the household.

2. If the child under age two is still alive at the listing time, there are three possibilities:

- a. Still alive and at same address for baseline survey: → interview

- b. Household moved but still lives in the same village: → find and interview

- c. Child under age two lives in another household that is in the target household list (and there is no other child under age two in this household):→ interview and add this household as a replacement.

3. If a household with children under age two has moved out of the village, replace the household.

4. Household replacement also applies in these cases:

- a. After four hours, the household still has not completed the interview. This could happen in households that contain only busy adults.

- b. A household with children under age two refuses to be interviewed. The supervisor must pay a visit to the household reported by interviewer and help solve any problems. If after the supervisor visit, the household still refuses, replace it.

8
- c. A household can be a duplicate if the head of household’s name, with the same characteristics, shows up more than once on the household list targeted to be interviewed in an enumeration area. In this case, only interview the household with the smallest number and replace the other household.

- d. A household cannot be reached after four hours. This could happen if all household members are out of town, or adult household members are too busy to meet.

- e. A household on the preprinted data listing is unknown to village authorities and villagers.

All replacements must be authorized by a supervisor.

# **5. DATA COLLECTION**

This section provides information about the datasets used in the evaluation and compares the characteristics of the treatment and control communities. We also examine the data for differential sample attrition across program and comparison villages.

A baseline survey was conducted in both treatment and control communities in August – September 2008. A total of 2,087 households in 160 sub-villages were interviewed. A health nutrition module to record recent illness for children under 5 years of age was also administered, providing a baseline sample of 2,353 children.

The follow-up data collection was conducted approximately 24 months later, between November 2010 and February 2011. The endline survey used the same field methodology as the baseline survey. The final sample size was 2,500 households. Of these, 1,908 were also surveyed in the baseline and so constitute our “panel sample” (see Table 1).

# **6. VALIDATION OF EVALUATION DESIGN**

# _6.1 Baseline Comparison of Means of Tests for Balance_

We use the 2008 baseline survey data to compare characteristics of treatment and control groups. Table 2 compares the means of selected key variables across treatment and control groups. Randomization aims to minimize systematic differences between the two groups. Table 2 shows that the means of the variables are similar in magnitude for the two groups and we cannot reject that they are equal for most of the variables. Thus the randomization successfully created groups comparable along observable dimensions. That is, mean differences between the treatment and control groups are not statistically different from zero. For the key outcome variables (household water and sanitation condition, as well as children’s health variables), balance is achieved. The 9
demographic and socio-economic characteristics are also similar across treatment and control groups. The baseline report provides tests of balance on a more extensive set of variables.<sup>6</sup>

# _6.2 Attrition in the Follow-up Data_

A legitimate concern in any follow-up survey involves the extent of sample attrition and the degree to which attrition is nonrandom. In the baseline, a total of 2,087 households were interviewed. Two years later, 1,908 households were successfully re-interviewed. Of the original 2,087 baseline household, 179 could not be contacted (86 households in the control group and 93 households in the treatment group). This figure translates into an overall attrition rate of 8.5 percent.

Despite the low rate of attrition found in the TSSM sample, nonrandom attrition could still lead to biased inferences. In this section, we investigate nonrandom attrition by examining the characteristics of households that left the sample. We examine whether means of outcome and control variables for the attriting households differ across treatment and control communities.

As Table 3 shows, the attriting household characteristics are similar across treatment and control groups. Only one difference is statistically significant at the 5 percent level: the attriting treatment group has fewer household members under five years of age than the attriting control group. A small number of differences are significant at the 10 percent level: the percentage of household heads with senior high school education is higher in the attriting treatment group ( _p=_ 0.057), as is dwelling ownership ( _p=_ 0.09) and the use of concrete walls ( _p=_ 0.05). All of these differences are relatively small and based on a small number of observations.

We also examine whether the households that dropped out of the sample differ from those that were found and surveyed for the endline. Table 4 suggests that the attriting households and panel households differ in some respects. Attriting households were less likely to have owned their dwelling at the baseline (73 versus 83 percent, _p=_ 0.001). They were also more likely to have a female head (10.6 versus 3.9 percent, _p<_ 0.000), and have more adults ( _p=_ 0.004) and fewer children between ages six and 10 (but no difference in the number of children under age two). Attriting households were also slightly wealthier ( _p=_ 0.098) and used more liquid petroleum gas (LPG) as cooking fuel ( _p<_ 0.05) _._ As discussed earlier, we added households to the endline sample to attain a sample size of 2,500 households. We did this to increase the power of our endline tests. Table 4 further examines whether the added households differ from the panel households. Columns 3 – 5 report the results of tests of difference in means. Only a small number of differences are statistically significant at the 10 percent level. A lower percentage of the new endline households (20 percent) received

> 6 See Annex 1 in _Scaling Up Rural Sanitation: Findings from the Impact Evaluation Baseline Survey in Indonesia_ , Water and Sanitation Project Technical Paper, November 2010; http://water.worldbank.org/publications/scalingrural-sanitation-findings-impact-evaluation-baseline-survey-indonesia.

10
BLT payments (that is, cash transfers from the government to poor households) than panel households (24 percent). We also observe a greater use of concrete walls and lower use of bamboo for new households. A higher proportion of new endline households had soap available at the handwashing station and had access to improved sanitation, but a lower proportion of them appropriately dispose of children’s feces. Thus, Tables 3 and 4 suggest that the attrition was mostly random with respect to treatment assignment so doesn’t lead our households in treatment communities to differ systematically from those in the control communities. Further, the panel sample does not appear to differ markedly from the original random sample so should be reasonably representative of the population.

# **7. PROGRAM EXPOSURE**

During the study period, trained facilitators went into the eight project districts and supported the local governments in conducting triggering and follow-up activities in the communities. Of the 80 treatment villages, the endline survey data established that 53 villages (66 percent) were triggered (see Table 5).<sup>7</sup> The table also reports data from an earlier, additional short survey of community heads conducted in May 2010, which generated very similar figures to the endline data. Monitoring data collected as part of the TSSM program reports a higher percentage of treatment villages (83 percent) received the triggering.

Some control villages in the sample did receive the treatment. This was largely as a result of district governments changing their target communities after the randomization plan had been agreed upon. The endline survey data suggests that 13.8 percent of the control villages were exposed to the program (TSSM monitoring data gives a lower figure, at 4 percent).

The extent to which districts adhered to the randomization varied considerably.<sup>8</sup> Jombang, Bondowoso, Blitar, and (to a lesser extent) Madiun performed well in this respect. Banyuwangi and Probolinggo performed less well, with triggering detected in only 50 percent of treatment villages. Situbondo and Ngawi adhered the least to the randomization plan, with treatment detected in only four of the 10 treatment villages (40 percent). The contamination of control communities does not vary markedly across districts.

Next, we look at the extent of exposure to TSSM by households in treatment communities. Table 6 presents evidence of triggering activities as reported by households in the endline survey.<sup>9</sup>

> 7This means that one or more of the four community leaders surveyed (village head, community head, health cadre, head of women’s organization, or other community leader) reported that the village had received a triggering. 8An independent study by Mathematica (2011) found similar results, in which program implementation varied heavily across the six districts they studied (three of the six districts are in our sample), depending upon capacity levels and cross-sectoral commitment to the program.

> 9This is in response to the question, “In the past two years, has there been a community event (such as a triggering/motivational talk) where sanitation was mentioned in this village?” Respondents were subsequently asked

11
Twenty-five percent of households in treatment villages reported that they knew of triggering activities being held in their community in the last two years, compared to 8.23 percent in control villages. The variation in these figures across districts is similar to that in Table 5. Bondowoso had the most households (42 percent) that were aware of the program in treatment villages. The table also indicates that participation in the triggering sessions differs across the districts.<sup>10</sup> While 30.13 percent of the households in treatment villages in Bondowoso attended the session, only 5 percent attended in Ngawi.

One of the key aims of the TSSM program is to increase awareness of good hygiene practices. To investigate this awareness, households were asked whether they had been exposed to sanitation campaigns through various media.<sup>11</sup> Figure 2a presents the results for any sanitation campaign (not necessarily TSSM). It shows that households received information about sanitation from many sources. Village health staff were an important conduit, as were TV campaigns. Exposure to sanitation messages was higher in treatment communities than in control communities, particularly via village staff (health staff and otherwise), billboards, and printed media.

Figure 2b shows the extent of exposure to messages that are explicitly about TSSM. Not surprisingly, the differences between treatment and control villages are much higher for TSSMrelated messages. There was significantly greater exposure to TSSM sanitation messages via TV, printed media, village notice boards, and village personnel. TSSM-related exposure was much lower than the exposure to any sanitation program. For example, in treatment villages, 13 percent of households had heard about TSSM via village health staff whereas 36 percent of households had heard sanitation messages connected with any sanitation campaign in this way.

> whether the triggering was a TSSM/STBM event. (STBM is another name by which TSSM is known.) Almost all respondents (89 percent) who answered that they knew of a triggering reported that it was a TSSM/STBM triggering. 10If respondents indicated that there had been a triggering in their village, they were asked, “Did you participate in this community event (triggering/motivational talk) where sanitation was mentioned?”

> 11Specifically, households were asked, “In the past two years do you remember seeing, hearing, or reading anything about an invitation/encouragement to stop open defecation and to use or build yourown toilet/latrine?” They were then asked whether this had been: a) a TV advertisement; b) radio advertisement; c) poster/printed media/wall calender/pocket calender/leaflet of products/merchandise/banner; d) video presentation; e) on a shop nameplate; f) on the village announcement board; g) from a health officer; h) from a triggering officer; i) from a sanitarian; j) from a village midwife; k) from the maternal health cadre; l) from the village/sub-district staff; or m) other source. If they answered yes to any of these categories, they were asked whether the message was from the program called TSSM or STBM.

12
**Figure 2. Household exposure to the sanitation program. (a) Percent of households that know about (any) sanitation campaign through various media and (b) whether it is TSSM related**

<!-- Start of picture text --> 50%<br>43.44<br>45% 41.71 35.47 34.88<br>40% *** *<br>35% 32.72 31.44<br>29.36 29.76<br>30%<br>25% 18.40 16.88<br>** 14.34<br>20% ***<br>14.17 15.23 ***<br>15% 13.29 7.61 9.94 11.60<br>10% 5.20. **<br>5% 2.16 2.56 2.24 2.96<br>0%<br>Control Treatment<br>(a)<br>50%<br>45%<br>40%<br>35%<br>30%<br>25%<br>20%<br>12.87 13.11<br>15% 12.63 ** ***<br>8.34<br>10% 9.31 6.08 6.20 *** 5.13<br>** ** 7.14<br>4.17 4.72 **<br>5% 2.55 3.44 2.03 3.32 3.57 0.60<br>0.72 1.15 2.68 0.00 **<br>0.64<br>0%<br>Control Treatment<br><!-- End of picture text -->

# **(b)**

13
Table 7 presents evidence of program exposure at the community level, in which the respondent is the village head (or in some cases a village staff member who is responsible for sanitation). Seventy percent of respondents in the treatment villages had heard about TSSM and more than 45 percent of the village heads in the treatment group attended the meeting and requested triggering activities. These numbers are all significantly higher than in control communities at the 1 percent level. Table 7 does, however, also show some contamination of control villages, with 11 percent of village heads in control villages reporting they had been to a triggering. Seventy-two percent of the village heads in the treatment villages are satisfied with the program.

The community section of the survey also asked village heads whether they helped households to gain quicker access to sanitation. In the treatment communities, 72 percent of the respondents stated that they did, mainly through building public toilets (45 percent), fundraising (17 percent), and providing lower-cost materials (9 percent).

In addition to surveying village heads, we interviewed community heads (sub-village heads or _kepala dusun)_ about program exposure _._ The lower panel in Table 7 reports the results. In the treatment villages, 53 percent of the _kepala dusun_ had heard about TSSM and 35 percent remembered the message (compared to 25 ( _p=_ 0.000) and 18 ( _p=_ 0.017) percent, respectively, in the control villages). Half of the treatment villages reported a triggering, with 38 percent reporting that the triggering was related to TSSM, and 38 percent of _kepala dusun_ attended the meeting. The figures are much lower for the control villages (statistically significantly different at the 1 percent level). Of the 38 percent of sub-village heads who attended the triggering session, only 13 percent found the facilitator very persuasive, with 77 percent reporting that the facilitator was somewhat persuasive.<sup>12</sup> Sixty-two percent reported that they were satisfied with the program, and 75 percent reported that sanitation has improved in the community post-triggering. In only 11 percent of cases has the community been declared ODF.

# **8. DESCRIPTIVE STATISTICS COMPARISON BETWEEN BASELINE AND ENDLINE**

Tables 8a and 8b provide descriptive statistics for key outcome variables along with control variables before and after program implementat **i** on. We first discuss the sanitation-related variables and then examine health outcomes.

# _8.1 Improved Sanitation and Toilet Construction_

Table 8a presents descriptive statistics for sanitation and hygiene-related outcomes. The baseline data indicate that around 43 percent of the households in both the treatment and the control

> 12 WSP (2012) found that poor-quality triggering decreased the likelihood of communities reaching ODF status and dampened communities’ desire to change behavior toward better hygiene practices.

14
communities had access to improved sanitation.<sup>13</sup> Access to improved sanitation had increased only very slightly, by about 1 percentage point in both treatment and control communities by endline. Figure 3 presents the percentage of households with no sanitation, unimproved sanitation, and improved sanitation in the baseline and endline by treatment status. It similarly shows small improvements in sanitation in both treatment and control communities.

**Figure 3. Households’ sanitation status**

<!-- Start of picture text --> 100%<br>90%<br>80%<br>70%<br>60%<br>No Sanitation<br>50%<br>Unimproved Sanitation<br>40% Improved Sanitation<br>30%<br>20%<br>10%<br>0%<br>Control-BL Control-EL Treatrment-BL Treatment-EL<br><!-- End of picture text --> Table 8a also shows that although more than 40 percent of the households in the sample had access to improved sanitation before the program, another 40 percent of households lacked access to any form of sanitation. These households, many of which are particularly poor, defecate in open spaces. The percentage of households with no access to sanitation of any sort declined by about 4 percentage points between the baseline and endline. Again the decline in treatment and control communities appears similar.

The endline data specifically asked households whether they had constructed a toilet in the past two years. This variable is reported in Table 8a and indicates that more toilets were built in treatment communities than in control communities. Approximately 16 percent of households

> 13 Improved sanitation is defined according to the WHO/UNICEF (2012) Joint Monitoring Programme for Water Supply and Sanitation (JMP; http://www.wssinfo.org). Improved sanitation facilities include: a) a flush toilet or latrine that flushes to a sewer, septic tank, or pit; b) a ventilated improved pit (VIP) latrine; c) a pit latrine with the pit well covered by a slab; or d) composting toilets. Shared and public toilets are considered “unimproved” regardless of their type.

15
reported having constructed a toilet in the last two years in treatment communities versus 13 percent in control communities. This difference is statistically significant ( _p_ =0.07) _._ To reconcile the finding that the percentage of households with access to improved sanitation did not increase in treatment villages relative to control villages with the finding that more toilets were built in treatment villages, Table 9 examines toilet building more closely. It shows that 275 (14.4 percent) of the 1,908 panel households reported constructing a toilet in the last two years, with 151 (16 percent) of these toilets built in treatment communities, and 124 (13 percent) built in control communities. Not all construction resulted in a shift from “unimproved” to “improved” sanitation—102 of these toilets were built by households who already had access to improved sanitation. Only 180 of the households that built toilets improved their sanitation as a result. The number of households that built a toilet and consequently improved their sanitation is only very slightly higher in treatment communities than in control communities (9.7 versus 9.2 percent) and not statistically significant ( _p_ =0.73). Note, however, that health may improve as a result of toilets being built by households that already had access to “improved sanitation.” For example, many of the households that had improved sanitation at baseline and had constructed a toilet by endline had upgraded to a porcelain latrine, which may be easier to keep clean and so more hygienic. Similarly, health benefits may accrue to households that don’t have access to “improved” sanitation but have nevertheless gone from defecating in the open to using a pit toilet (without a concrete slab).

The endline survey also asked respondents when the toilet was built. Figure 4 reports the percentage of households that built a toilet prior to the endline survey, which was conducted between November 2010 and February 2011. Overall, a high percentage of households reported that they built their toilet between 12 and 24 months prior to the endline survey—almost 40 percent of the treated households reported that their toilet facility was built 24 months prior to the endline survey. The figure is slightly smaller for households in the control group (32 percent).

16
## **Figure 4. When did households build their toilets?**

<!-- Start of picture text --> 45%<br>39.74<br>40% 38.71<br>35.10<br>35%<br>32.26<br>30%<br>25%<br>20%<br>15%<br>Control Treatment<br>10%<br>7.26<br>5.65<br>4.64 4.64 4.84 4.64<br>5% 3.23 2.65 2.42 2.65 2.65 2.65<br>2.42<br>0.81 0.81 0.81 0.66 0.81<br>0%<br><= 1 2 3 4 5 6 7 8 10 11 12 24<br>month<br>Number of months prior to endline survey<br><!-- End of picture text -->

# _8.2 Open Defecation_

Households were also directly asked whether household members ever engage in open defecation (on a daily, seasonal, or occasional basis). This question was asked regardless of the type of sanitation facility households had access to and where they reported they normally defecate. These data were reported separately for male household members, female household members, and children under five years of age, and are presented in Table 10. The table shows that 51 percent of households reported that at least one household member engages in open defecation. Women engage in open defecation slightly less often than men and children. Most open defecation in East Java occurs in rivers (95.36 percent of households that do not have access to sanitation report defecating in rivers). In villages that are within 20 minutes of a river, 54 percent of households had at least one member who defecates in the open at least occasionally. The figure is much lower (13 percent) for households in villages that are more than 20 minutes from a river. The fraction of households reporting that at least one member engages in open defecation is systematically lower in treatment communities than in control communities. The percentage of households reporting that a household member defecates in the open is 4.4 percentage points lower in treatment villages (significant at the 5 percent level). Among 17
communities on rivers, open defecation by any household member is 6 percentage points lower in treatment communities than in control communities. The declines in open defecation are driven by declines in open defecation by people who have access to a toilet. Twenty-four percent of households that have access to a toilet reported sometimes defecating in the open; this figure is 4 percentage points lower in treatment communities than in control communities (26 versus 22 percent).

# _8.3 Handwashing_

Improvements in good hygiene practices, especially handwashing with soap at critical times (before handling food and after defecating) can also help prevent childhood diarrhea (Ejemot et al., 2008) and other illnesses such as acute lower respiratory infection (Aiello et al., 2008). Although TSSM’s message focuses on improvements in sanitation as the main vehicle for improving child health, the benefits of handwashing with soap were also discussed. The endline survey collected information on households’ handwashing facilities and behavior.

Table 8a shows that almost 100 percent of respondents reported that they wash their hands after defecating. Of these, 67 percent reported having a handwashing station where they usually wash their hands. Observation by the survey enumerators of the handwashing station revealed that water was available at about 96 percent of the stations, and soap and water were available at about 58 percent of stations. These figures had increased from the baseline but there was no significant difference between treatment and control communities.

# _8.4 Drinking Water_

Drinking water quality is another important determinant of child diarrhea and parasitic infection. The WHO defines improved drinking water sources as tap water inside the house and outside the house, protected dug well, boreholes (includes tube wells), protected springs, and rainwater collection (WHO, 2012). Table 8a shows that a large majority (more than 87 percent) of households had access to improved sources of water in both the baseline and the endline. There is no difference between treatment and control communities with respect to access to improved sources of drinking water. Note that we do not expect differences in drinking water outcomes given that this was not part of the treatment; however, it is interesting to note that differences in drinking water cannot be driving the health impacts we observe.

# _8.5 Attitudinal Change_

TSSM seeks to change sanitation-related behavior via attitudinal change. The surveys collected information on people’s attitudes about sanitation and the extent of their knowledge of the causes of diarrhea. Table 11 compares attitudes to open defecation in treatment and control communities. These questions were asked only in the endline survey. The vast majority of households (96.1 percent) agreed or strongly agreed with the statement that having a toilet benefits the community 18
as it reduces environmental pollution. Similarly, an overwhelming majority (92 percent) agreed or strongly agreed with the statement that having a toilet protects their family against illness such as diarrhea. Further, 78.4 percent of households reported that having a toilet reduces the probability of being a topic of gossip, and only 6.9 percent of households agreed with the proposition that open defecation is acceptable because their ancestors practiced it. Nevertheless, the figures also show a tolerance of those who need to engage in open defecation. For example, 31.3 percent reported that it is acceptable to practice open defecation if you don’t have a toilet, and only 34 percent reported that those who practice open defecation will be socially shunned. Table 11 disaggregates this information by treatment and control communities. There are no significant differences across the communities. Thus, it seems that knowledge of the health consequences of poor sanitation is relatively high across East Java, even in the absence of the program.

Table 12 examines caregivers’ perceptions of the causes of diarrhea. These data were collected in both the baseline and endline surveys. Almost all caregivers (approximately 97 percent) thought that diarrhea is caused by eating dirty food or drinking dirty water. Lower proportions thought that it is caused by not washing hands (about 82 percent). About 86 percent reported environmental contamination as a cause, although only 72 percent believed that it is a consequence of others defecating in the open, and a smaller percentage (60 percent) believed it is a result of others defecating in rivers. The belief that diarrhea is caused by unclean food and water increased in both treatment and control communities but by significantly more in treatment communities. This is also the case for the belief that diarrhea is caused by an unclean environment. Increases in treatment villages relative to control villages in the belief that diarrhea is caused by some vaccines and exposure to the sun are more difficult to explain. All of the differences are small, which is perhaps to be expected when so many households seem to have been relatively well-informed about the causes of diarrhea before program implementation.

Households were also asked how much it would cost to build or improve their own toilet and to state the maximum that they would be willing to pay. Fifty-seven percent of households reported a figure equal to the figure they had reported for constructing the cheapest latrine possible. However, 38 percent of households reported a higher figure, indicating that they were only interested in building a more elaborate toilet. Figure 5 shows the relationship between perceived affordability and household income. Many households with household income below the mean reported that the amount they would have to pay is many times the maximum amount they were willing to pay. This is less common among higher income households.

19
**Figure 5. Affordability to build or improve a toilet by household income**

<!-- Start of picture text --> Affordability by Household Income<br>0 20 40 60<br>Total household income (mill Rp)<br>10<br>8<br>6<br>4<br>2<br>Estimated Cost/ Max Willing to Pay<br>0<br><!-- End of picture text --> The surveys also collected information on respondents’ level of satisfaction with their sanitation facility. One might expect treatment communities to become less satisfied with having no sanitation facility as a result of the program. The data show slightly more dissatisfaction among households without access to a toilet in treatment communities but the difference is small and statistically insignificant (29.2 percent in treatment communities versus 28.1 percent in control communities, _p=_ 0.74). Satisfaction with open defecation remains high: 55.6 percent of households that don’t have access to sanitation reported that they were satisfied (38.8 percent) or very satisfied (16.8 percent) with their sanitation. This reflects the custom of defecating in rivers, which is thought to be hygienic because the waste is washed away. East Java is rich in rivers and 117 of our 160 communities (73 percent) are within a 10-minute walk from a river. Note that rivers are generally not used as a source of drinking water in Indonesia. Hence, a greater awareness that diarrhea is caused by dirty drinking water is not necessarily inconsistent with high levels of satisfaction with defecating in rivers.

# _8.6 Obstacles to Latrine Construction_

Households that indicated they would like to build a latrine were asked what they consider to be the main obstacle to building one. Figure 6 presents the data from the endline survey. As reported in the baseline survey, the main obstacle was the cost of construction. TSSM’s supply20
side intervention involved training local masons<sup>14</sup> to build the kind of latrines that the local populations desired. Some districts also instituted schemes to make latrines more affordable (by providing access to credit, instituting rotating saving schemes with savings to be used for latrine construction ( _arisan-jamban_ ), and mobilizing community labor to build latrines).

The implementation of the supply side of TSSM, however, lagged behind that of the demand side and was implemented on a much smaller scale. Table 13 shows how much households estimated it would cost to construct the cheapest possible latrine. On average, households estimated it costs approximately Rp 1.2 million to build a latrine (US$135). This is about onethird of the annual per capita income for the average household in our sample. TSSM estimates of the actual cost to install a latrine with a slab (exclusive of the superstructure) are in the range of US$50 to US$90 (WSP, 2012). There is no statistically significant difference in the costs reported in treatment communities and control communities. The majority of households (64 percent) reported that these costs rose in the last 12 months, in both treatment and control communities. There are also no differences between treatment and control communities with respect to access to materials and tradesmen. Slightly more people in control communities reported that they do not know who to contact to obtain materials and tradesmen (4.3 versus 3.1 percent, _p<_ 0.05). Slightly more people in treatment communities reported that the costs are affordable, but this difference is not significant.

**Figure 6. Main reason for not building a toilet**

<!-- Start of picture text --> Tenancy issues 0.11<br>Permit issues 0.22<br>Too complex 0.34<br>Water not available 0.45<br>No one to build 0.67<br>Soil condition 0.90<br>No material 1.35<br>Satisfied with current facility 1.91<br>No savings 3.03<br>Others 5.38<br>Limited space 9.87<br>Other competing priorities 28.92<br>High cost 46.86<br>0 5 10 15 20 25 30 35 40 45 50<br><!-- End of picture text -->

> 14Later, this was replaced by capacity building of more full-fledged sanitation entrepreneurs who employ masons.

21
To summarize, the results to this point show that the number of toilets built in treatment communities since the program’s implementation was higher (3 percentage points) than in control communities. This is a 31 percent increase in the rate of toilet construction in treatment communities relative to control communities. There is evidence of declines in occasional open defecation by households that have access to a sanitation facility. Significantly fewer households in treatment communities reported that one or more household members defecate in the open (4.4 percentage points fewer). Knowledge of the health benefits of improved sanitation seems to be high across East Java, even in control communities. And the program is associated with increased acknowledgement that unclean food and water leads to diarrhea. However, defecating in rivers is still an accepted practice. The main obstacle to constructing facilities as reported by households seems to be the cost.

- _8.7 Prevalence of Diarrhea, Acute Lower Respiratory Infection (ALRI), Helminth Infections, and Acute Respiratory Infection (ARI)_

Now we turn our attention to health outcomes. Table 8b presents the summary statistics for the health variables. Diarrhea was defined using the WHO definition (three or more stools per day and the stools were loose or watery, or blood and/or mucus is visible in the stool (Baqui et al., 1991) based on symptoms reported by the child’s caregiver. There was a decline in diarrhea prevalence between the baseline and endline surveys.<sup>15</sup> Diarrhea prevalence was balanced in the baseline. In the endline, diarrhea prevalence in the treatment group was lower than that in the control group, regardless of the recall period used. Using a recall period of seven days, 2.4 percent of children under the age of five in the treatment group reported having had diarrhea in comparison to about 3.8 percent in the control group. The same pattern is observed if we use the shorter recall period of two days. These differences are statistically significant.<sup>16</sup> Soil transmitted helminthes ( _Ascaris lumbricoides, Trichuris trichuria,_ hookworm) are estimated to infect about one-third of the world’s population, and a significant share of infection is born by preschool-aged children (Albonico et al., 2008). Helminth infections can lead to iron-deficiency anemia, protein-energy malnutrition, abdominal pain, and listlessness (Adams et al., 1994). To examine this, stool specimens were collected and analyzed at endline from a random subsample of 2,330 children in the 80 villages. Each child in the parasitological sample was given a plastic container (a stool specimen container, with a spoon attached to the lid) and asked to provide a stool sample. The container had 15 ml of preservative (formalin) in it, unless the child was reported to have diarrhea, in which case there was 30 ml of formalin in it. Caregivers were told

> 15 This decrease at least in part reflects the aging of the children as these are figures for children who were present during both the baseline and endline surveys. Diarrhea prevalence is higher in children under the age of two. It may also be due to the different timing of the two surveys. The endline survey was conducted in the rainy season, whereas the baseline was conducted at the end of the dry season.

> 16 Shorter recall periods may be a more accurate measure of diarrhea prevalence because they involve less recall error.

22
that specimens must not be contaminated with urine or water. They were also told that the preferred time for specimen collection was early in the morning. Caregivers were instructed to scoop enough stool to fill approximately one-third of the container (about 5 grams for solid stool and 10 ml for liquid stool, which was about two full scoops), screw the cap on tightly, and shake well. The sample was then transferred to a lab. Following Katz (1972), reported infection intensity was proxied by worm eggs per gram of stool.

Table 8b reports the results from the parasitology analysis. Worm infection was relatively low in our sample, with only 4 percent of the surveyed children testing positive for at least one of the following: worms ( _Ascaris lumbricoides, Trichuris trichuria_ ) and hookworms ( _Ancylostoma duodenale_ and _Necator americanus_ ). We found no significant difference in the probability of having any eggs between the children in the treatment and control groups. Children in the treatment groups appear to have lower infection intensity than those in control groups. However, the difference is not statistically significant. Table 8b also provides information on the prevalence and intensity of infection by organism. Prevalence is highest for _Ascaris_ (3 percent). Hookworm prevalence is 0.5 percent, and _trichuris_ prevalence is 0.1 percent _._ We compare infection prevalence categorized by low, moderate, and high infection intensity using the WHO (2002) thresholds for fecal egg count per gram (epg).<sup>17</sup> As before, we find no significant difference in infection intensity for each organism in the treatment and control groups.

Acute respiratory infections (ARI) are among the leading causes of death in children under five years old. Following the WHO definition, a child is identified as having ARI when he or she presents with a persistent cough or difficulty breathing. A meta-study conducted by Roth et al. (2008) found that throughout the world, 1.9 million children died from ARI in 2000; 70 percent of these deaths were in Africa and Southeast Asia. Acute lower respiratory infection (ALRI) is a more serious condition and is diagnosed when a child has a constant cough or difficulty breathing (ARI) and a raised respiratory rate. A number of factors affect ARI and ALRI rates in young children, including malnutrition, lack of breastfeeding, and the incidence of other diseases that affect susceptibility. The child’s environment (such as crowding and air pollution) can also affect the risk of getting ALRI. Better sanitation in the form of frequent handwashing can reduce the spread of the infection from person to person.

As Table 8b shows, using either a seven- or two-day recall period, ARI and ALRI prevalence are slightly higher for children in the treatment group than for children in the control group, although the differences are small and not statistically significant. ALRI prevalence in the sample is low (around 2 percent of children were found to have had ALRI in the past seven days).

> 17 The infection thresholds for _Ascaris lumbricoides_ are low: 1–4,999 eggs per gram (epg); moderate: 5,000–49,999 epg; and heavy: >50,000 epg. For _trichuris trichuria_ , infection thresholds are low: 1–999 epg; moderate: 1,000– 9,999 epg; and heavy: >10,000 epg. For hookworm, thresholds are low: 1–1,999; moderate: 2,000–3,999; and heavy: >4,000 epg (WHO, 2002).

23
Table 8b shows similar comparisons for a range of other symptoms of child illness. Changes in health symptoms that are unlikely to be affected by the treatment can be used to confirm that the observed differences between the treatment and control communities are not attributable to systematic reporting bias. Bruising/abrasions and itchy skin/scalp are two symptoms that are unlikely to be affected by the intervention. Table 8b shows that the differences in these symptoms between control and treatment communities are small and statistically insignificant.

Indeed, most of the reported symptoms show no difference between the control and treatment groups in the endline. However, the prevalence of mucus or blood in the stool is significantly lower for children in the treatment group (0.8 percent _cf._ 2.0 percent). Refusal to eat is also lower in the treatment group. Apart from these symptoms, only the prevalence of nasal congestion differs between the control and treatment communities. It is higher in the treatment communities.

# _8.8 Anemia, Anthropometric, and Developmental Measures_

Poor sanitation leads to diarrhea and diarrhea can increase the prevalence of anemia because it reduces the absorption of nutritional intake, including the absorption of iron, which is used in the production of hemoglobin. Any child with hemoglobin levels below 110 g/L is considered anemic as per WHO guidelines for children under five years of age (UNICEF, WHO, and United Nations University, 2001; WHO, 2011). Hemoglobin levels rose during the study period, consistent with the children having aged between baseline and endline. This translates into a much lower proportion of children with anemia at the end of the study period than at the beginning (44.2 versus 71.2 percent). However, there is no evidence of great declines in treatment communities vis-à-vis control communities. There is no significant difference in hemoglobin levels and anemia between the control and treatment groups.

Anthropometric measurements (weight, height, arm and head circumference, body mass index) were taken for all children under five years old at the time of the baseline in the surveyed households. Child anthropometric measurements provide useful information about nutritional intake and so are adversely affected by illnesses such as diarrhea. Weight (conditional on height and gender) typically varies in the short-term so is used as a measure of current health status whereas height, given age and gender, is an indicator of longer-term health and welfare (Thomas, et al., 1991).

To assess the child’s growth and general nutritional status, we used a standardized age- and gender-specific growth reference based on the WHO (2006, 2007) standard to calculate _z_ -scores for a) weight-for-height, b) height-for-age, c) weight-for-age, d) body-mass-index, e) head circumference-for-age, and f) arm circumference-for-age.<sup>18</sup> All of the child growth measures

> 18 For example, a _z_ -score for height subtracts from the child’s height, the median height in the reference population, for a child of the same gender and age in months, and divides by the standard deviation of height in the reference population, also for a child of the same gender and age in months. A weight-for-height _z_ -score is defined in an analogous manner, except that the standardization is done using the reference population median and standard

24
(with the exception of upper arm circumference in the baseline) tended to be lower than the population mean using the WHO standard (i.e., have negative _z_ -scores). All of these measures decreased between the baseline and the endline, reflecting the aging of the cohorts and the growth faltering that occurs during the first two years of life (Victora et al., 2010). There are no statistically significant differences in the anthropometric measures between the treatment and control groups.

# _8.9 Child Development_

Both the baseline and endline surveys included an “Ages and Stages” questionnaire, which measures child development across five domains: communication, gross motor skills, fine motor skills, problem solving, and personal-social development. At the baseline this questionnaire was administered to parents or caregivers of children ages 3 – 24 months. In the endline survey it was extended to children up to 60 months of age. This report focuses on the communication, motor skills, and personal-social development. These measures are of interest because a child’s health status might affect his or her development. The questions asked about each child were selected according to the child’s age in months. To make a comparison, the child development index for each skill is standardized by calculating _z_ -scores.<sup>19</sup> Table 8b shows greater improvement in communication (significant at the 10 percent level) for children in treatment communities.

To summarize, a comparison of raw outcomes in treatment and control communities shows greater toilet construction in treatment communities and declines in open defecation. In line with this, there is evidence of decreases in diarrhea prevalence in treatment communities, relative to control communities. The prevalence of mucus or blood in the stool has also decreased. Health outcomes further along the causal chain appear to have been largely unaffected by the TSSM program.

# **9. ESTIMATION STRATEGY AND RESULTS**

In this section, we estimate the “intention-to-treat” (ITT) of program impact. The ITT approach estimates the program impact from a comparison of those communities to which the program has been offered, regardless of whether they actually receive it, with control communities. The ITT allows us to determine a program’s average impact on the population it is targeting and is often used in the context of randomized controlled trials (Duflo et al., 2008). The ITT estimator is computed by comparing the mean outcome for the treatment group to that for the control group, which can be estimated using the following regression equation:

> deviation of weight for children of a given gender and height. The WHO standards use an international reference population.

> 19 _Z_ -scores standardize the measures so they have a mean of zero and a standard deviation of 1. The different measures are thus expressed in the same scale and so can be compared across different age groups.

25
where _Yij_ is the outcome for observation _i_ in village _j_ conditional on treatment; _Tij_ is the treatment dummy, which is 1 for the households or children in the treatment group, and 0 otherwise; and _εij_ is the error term that is correlated within villages, given the cluster-RCT design.

If one only controls for treatment status on the right-hand side of the equation, ITT generates the same results as the differences in means presented in the previous section. The ITT regression approach, however, also allows us to control for other variables. In the specifications reported next, we add a set of pairwise sub-district dummy variables ( _γj_ ) to control for sub-district effects given that the randomization was assigned such that in each of the participating districts, 10 (treatment and control) pairs of villages from the same _kecamatan_ (sub-district) were chosen.

In some specifications, we also add lagged dependent variables (i.e., outcomes as observed in the baseline _Yij[BL]_ ) and a vector of household characteristics known to influence sanitation practices and health outcomes ( _Xi_ ) as additional right-hand side controls.

Here, _β_ is the parameter of interest and represents the effect of the TSSM program on sanitation/health outcomes, conditional on **X** and the baseline outcome variables; and _τ_ reflects the degree to which health/sanitation outcomes are correlated over time. All specifications also allow for village-level clustering of the standard errors.

Finally, in some specifications we estimate the model conditional on whether households had sanitation at baseline. Doing so enables us to examine the effect of behavioral change on those that already have toilets (using them more); as well as those who do not have toilets (using public facilities more). In addition, we further disaggregate households at baseline who did not have sanitation facilities by poor and non-poor households. This can help us better understand whether there was differential construction of toilets and/or behavioral change due to income differences. We can also detect any health improvements resulting from the behavioral changes toward better hygiene practice.

# **Results**

# _9.1 Program Effects on Sanitation and Hygiene Behavior_

Table 14 presents the estimated impact of the TSSM program on sanitation. It presents means of the dependent variable in the control group in column 1 as an aid to interpreting the magnitude of the effects. Column 2 reports the result of estimating Equation (1) with pairwise sub-district fixed effects and shows a moderate increase in the rate of toilet building in the last two years in 26
the treatment households (4 percentage points higher for households residing in treatment districts compared to the control group). Column 3 restricts the sample size to panel households. The point estimate still shows greater toilet building in treatment communities, although it is now slightly smaller. The point estimate remains similar in magnitude and significant once we control for baseline sanitation (column 4)<sup>20</sup> and various exogenous controls (columns 5 and 6). Columns 7 and 8 show the estimate conditioning on whether households had a toilet at baseline. TSSM is associated with a 4 percentage point significant increase in toilet construction among households that did not have a toilet at baseline (column 7). The increase in toilet construction is smaller (1 percentage point) for households that already had a toilet at baseline and is not statistically significant (column 8).

The other rows in the table show the same analysis for improved sanitation, where a household normally defecates, whether any householder defecates in the open even occasionally, whether the household correctly disposes of children’s feces, and whether the household has access to improved drinking water. We find that access to improved sanitation increased significantly in treatment communities for those who did not have a toilet at baseline. Open defecation also decreased significantly in treatment communities among households that had no sanitation at baseline (5.8 percentage point reduction from a 48 percent base). We see no such decrease among those who had access to a toilet at baseline. We also find a significant improvement in the disposal of children’s feces among households that had no sanitation at baseline. We find no improvements in behavior among those who did have sanitation at baseline. This suggests that the behavioral changes were driven by toilet construction.

Table 15 reports the results for handwashing behavior. Results are presented in the same order as in Table 14. There is no evidence of improvements in handwashing behavior, which is not surprising given that handwashing was not a key component of the intervention. The only significant coefficient suggests that treatment is associated with a decline in water availability at handwashing stations.

The main obstacle to constructing a toilet reported by households was the cost. Credit constraints likely limited households’ ability to build a toilet. To investigate this, we estimate the model separately for poor and non-poor households. A household is defined as poor if they are in the bottom quintile in terms of the value of their non-land assets (less than Rp. 1,975,000). These households are actually “particularly poor.” Note that at baseline 36 percent of our sample fell below the 2008 rural Indonesian poverty line of Rp 161,831 per capita per month. Our results suggest that the treated non-poor households are building toilets, not the poor (Table 16). This suggests that provision of credit or subsidies to the particularly poor households may work to increase the project impact. Table 16 also shows that the program increased access to improved

> 20 For the building toilet variable, we used access to improved sanitation in the baseline as our baseline control variables.

27
sanitation for non-poor households that didn’t have access to any sanitation at baseline. This group also showed improvements in the disposal of children’s feces.<sup>21</sup>

# _9.2 Program Effects on Child Health Outcomes_

Table 17 reports the ITT effects for child health outcomes. Consistent with the data presented in Table 8b, we find a statistically significant reduction in diarrhea prevalence (seven- and two-day prevalence). Specifically, being in a treatment community is associated with diarrhea prevalence being lower by approximately 1.3 percentage points from a 4.6 percent base. This is the case across all specifications on the panel observations (columns 3 – 6) and for seven- and two-day prevalence. This is a large effect—about a 30 percent (44 percent) reduction in seven-day (twoday) diarrhea prevalence. The fact that the estimated program effect changes only slightly in magnitude as we add controls is unsurprising given that the TSSM survey provides a balanced sample.<sup>22</sup> Columns 7 and 8 show that this is being driven by decreases in diarrhea among households that did not have sanitation at baseline.

We also find some evidence that parasitic infection intensity for both _Ascaris_ and _Trichuris_ , as proxied by number of egg worms per milligram, is lower in treatment communities. However, this result is not statistically significant over the whole sample.<sup>23</sup> Having blood or mucus in the stool is also significantly lower in treatment communities by about 1 percentage point (57 percent) as well as refusal to eat for children in treatment communities.

> 21 Additional one-tailed tests of significance (not reported in Table 16) show a significant decrease at the 10 percent level in the practice of open defecation for the treated non-poor households with no sanitation at baseline.

> 22 The estimate in column 2 uses the entire endline sample (not restricted to panel households) and although the point estimate is negative, it is weakly significant for seven-day prevalence ( _p_ <0.1), while it is statistically insignificant for two-day prevalence. Note that the use of the full sample involves adding children under the age of two, whereas all children in the panel sample are over age two as they had to have been born at the time of the baseline. This is shown in Figure 7. If we re-estimate these regressions only on children under age two, we find no treatment effect on diarrhea prevalence. Hence, the weakly significant and insignificance of the full sample results may be being driven by the younger age of the children. It may be that the treatment has a reduced impact on diarrhea of very young children because children under age two are unlikely to use a new facilitation facility themselves. In contrast, children between ages two and five would be taking themselves to the toilet so may be more likely to directly benefit from new facilities.

> 23 We also estimated the model with the dependent variable being the prevalence of each of _Ascaris, Trichuris_ , and hookworm categorized using the WHO (2002) thresholds into low, moderate, and heavy infection. We find no significant treatment effect.

28
**Figure 7. Age distribution comparisons for panel and non-panel children**

<!-- Start of picture text --> Age distribution for kids at Baseline Age distribution for kids in the endline sample<br>N=2343 N=2638<br>0 20 40 60 0 20 40 60<br>agemonths_BL agemonths<br>Age distribution for kids in the endline sample Age distribution for kids in the endline sample<br>panel kid; N=1924 non-panel kid; N=714<br>20 30 40 50 60 70 0 5 10 15 20 25<br>agemonths agemonths<br>.05 .04<br>.04<br>.03<br>.03<br>.02<br>Density Density<br>.02<br>.01<br>.01<br>0 0<br>.06 .08<br>.06<br>.04<br>.04<br>Density Density<br>.02<br>.02<br>0 0<br><!-- End of picture text --> Columns 8b and 8c estimate the health impacts separately for poor and non-poor households that did not have sanitation at baseline. Diarrhea prevalence decreased in both of these subgroups, although by more for poor households. This is curious because non-poor households constructed more toilets. It may be that wealthier households are partially protected by other aspects of their environment that reduce the probability of getting diarrhea—for example, cleaner foodpreparation areas. Note, however, that a one-tailed test of significance (not reported in the table) shows a weakly significant (at the 10 percent level) reduction in the seven days of diarrhea prevalence for non-poor household with no sanitation at baseline. A one-tailed test of significance also finds a decrease in the occurrence of mucus or blood in the stool samples for 29
this group. Further, parasitic infection ( _Alascaris_ ) decreased significantly in treatment communities for the non-poor who had no sanitation at baseline.<sup>24</sup> Congestion and breathing difficulty are significantly higher in treatment communities for nonpoor households that did not have sanitation at baseline, for which we have no explanation.

When examining anthropometric and developmental impacts, we do not use _z_ -scores because we are concerned about the propensity for measurement error in age (which is considerable in Indonesia) to affect the results. Instead, we use the raw measures and include age dummy variables (defined in terms of three-month age blocks) on the right-hand side. Increases in weight, height, and weight-for-height were detected for non-poor households that had no sanitation at baseline (again, these are the households that were most likely to build a toilet as a result of the program). Although Table 17 does not indicate significant effects for poor households with no sanitation at baseline in terms of weight and height, it shows that treatment is associated with a higher BMI for this group. This may be an artifact of lower average height in treatment communities (insignificantly so). The estimates for head circumference are insignificant. Arm circumference indicates a negative treatment effect but this becomes insignificant once we break the sample down by whether or not households had sanitation at baseline.

Communication skills are positively impacted and social skills negatively impacted in some specifications but overall we find no systematic program impact on the development indicators.

# _9.3 Discussion of Results_

We find evidence of increased toilet construction and reduced open defecation in treatment communities relative to control communities. These improvements were largely among households that had no sanitation at baseline and were not “particularly poor” and so could afford to construct a toilet. We also find improvements in health among these households, although some health benefits seem to extend to poorer households that also had no sanitation at baseline.

The estimated reductions in diarrhea are large—1.4 percentage points or 30 percent—whereas the increases in toilet construction and reduction in open defecation practices are more moderate. Two questions that naturally arise are how plausible are the diarrhea results and are they truly a consequence of the program. The results disaggregated by sanitation status support the argument that the health improvements are a consequence of the program. The health improvements largely accrue to those who did not have sanitation at baseline and so benefitted the most from toilet construction.

> 24Note that if we disaggregate by proximity of the village to the river as in Table 10, we find that the decreases in diarrhea are driven by decreases in treatment villages that are on rivers. This is consistent with Table 10’s finding that open defecation decreased most in villages on rivers.

30
We cannot rule out that the difference in caregiver-reported diarrhea prevalence between groups could be an artifact of biased outcome reporting. However, we do not observe differences between groups in falsification symptoms that could be subject to similar bias. Moreover, the results from the fecal samples (arguably more objective data than caregiver-reported diarrhea prevalence) show benefits accruing even more narrowly to non-poor households that had no sanitation at baseline, which are the group most likely to have built toilets as a result of the program. The weight and height improvements are also restricted to this group.

Positive externalities are another potential explanation for the large estimated impact on diarrhea prevalence. Perhaps only small improvements in sanitation are needed to give rise to large health effects given the benefits that flow across the community as a result of a less contaminated environment. The benefits of a household’s new toilet extend beyond that household’s. This is an area we have not fully explored in this report but is an area for future research.

# **10. CONCLUSION**

In conclusion, knowledge of the link between poor sanitation and health seems to be high in East Java even in control communities. Nevertheless, open defecation is still tolerated and many households believe that other people defecating in the open has no bearing on their own household’s health. Defecating in rivers is still a relatively attractive option for many households. A qualitative study conducted by WSP (2012) found that communities located near rivers are less likely to achieve ODF status. Households in these areas considered defecating in the river superior to using a pit latrine. This suggests that specific strategies focused directly at communities that are in close proximity to water bodies might be needed.

Nevertheless, we find that TSSM is associated with sanitation improvements. More households built toilets in treatment communities (3 percentage points higher) than control communities. This is a 31 percent increase in the rate of toilet construction in treatment communities relative to control communities. Further, fewer households reported that household members ever defecate in the open in treatment villages (4.4 percentage points lower than in control communities). The sanitation improvements were largely driven by toilet construction by non-poor households. Sanitation improvements were more limited for “particularly poor” households. Poor households are more likely credit-constrained and are thus less able to build toilets and so improve their access to sanitation.

In terms of health outcomes, we detect significant declines in the prevalence of diarrhea among young children (between two and five years old). Diarrhea prevalence decreased by 1.5 percentage points more (a decline of 30 – 45 percent depending on whether we examine sevenor two-day prevalence) in treatment communities than in control communities. This is a large impact. We also find decreases in the intensity of parasitic infestation, and increases in height and weight among non-poor households that had no sanitation at baseline.

31