metadata: authors: 0: author: Hasan,Amer
1: author: Jung,Haeil
2: author: Kinnell,Angela
3: author: Maika,Amelia
4: author: Nakajima,Nozomi
5: author: Pradhan,Menno Prasad
count: Indonesia
docna: 0: docna: Contrasting Experiences : Understanding the
            Longer-Term Impact of Improving Access to Preschool
            Education in Rural Indonesia
docty: Policy Research Working Paper
abstracts: cdata!: This paper examines the longer-term
            impact of a project that expanded access to playgroup
            services in rural Indonesia. It compares the outcomes of two
            cohorts of children who were exposed to the same
            intervention at different points in time. One cohort was
            eligible to access playgroups during the first year of a
            five-year project cycle, beginning at age four. The other
            cohort became eligible to access these services during the
            third year, beginning at age three. The younger cohort was
            more likely to be exposed to playgroups for longer and at
            age-appropriate times relative to the older cohort. The
            paper finds that enrollment rates and enrollment duration in
            preprimary education increased for both cohorts, but the
            enrollment effects were larger for the younger cohort. In
            terms of child development outcomes, there were short term
            effects at age five that did not last until age eight, for
            both cohorts. The data reveal that the younger cohort had
            substantially higher test scores during the early grades of
            primary school, relative to the older cohort. To unpack why
            the two cohorts experienced different longer-term outcomes,
            the paper provides evidence of changes that transpired in
            the operating conditions of the playgroups over time.
geo_reg_key: 517198,517191
repnme: repnme: Contrasting Experiences : Understanding the
            Longer-Term Impact of Improving Access to Preschool
            Education in Rural Indonesia
display_title: Contrasting Experiences :
            Understanding the Longer-Term Impact of Improving Access to
            Preschool Education in Rural Indonesia
listing_relative_url: /research/2019/11/31566816/contrasting-experiences-understanding-longer-term-impact-improving-access-preschool-education-rural-indonesia
publishtoextweb_dt: 2019-11-19T00:00:00Z
origu: Education EAP (HEAED)
no_of_pages: 42
author_keys: 0: author_key: 807987
1: author_key: 1201240
2: author_key: 1278380
3: author_key: 1428753
4: author_key: 1320512
5: author_key: 123205
model_extractions: Policy Research Working Paper 9060

# Contrasting Experiences

Understanding the Longer-Term Impact of Improving Access to Preschool Education in Rural Indonesia _Amer Hasan Haeil Jung Angela Kinnell Amelia Maika Nozomi Nakajima Menno Pradhan_ Education Global Practice November 2019
Policy Research Working Paper 9060

# **Abstract**

This paper examines the longer-term impact of a project that expanded access to playgroup services in rural Indonesia. It compares the outcomes of two cohorts of children who were exposed to the same intervention at different points in time. One cohort was eligible to access playgroups during the first year of a five-year project cycle, beginning at age four. The other cohort became eligible to access these services during the third year, beginning at age three. The younger cohort was more likely to be exposed to playgroups for longer and at age-appropriate times relative to the older cohort. The paper finds that enrollment rates and enrollment duration in preprimary education increased for both cohorts, but the enrollment effects were larger for the younger cohort. In terms of child development outcomes, there were short term effects at age five that did not last until age eight, for both cohorts. The data reveal that the younger cohort had substantially higher test scores during the early grades of primary school, relative to the older cohort. To unpack why the two cohorts experienced different longer-term outcomes, the paper provides evidence of changes that transpired in the operating conditions of the playgroups over time.

This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at ahasan1@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Contrasting Experiences: Understanding the Longer-Term Impact of Improving Access to Preschool Education in Rural Indonesia**

# **Authors***

Amer Hasan (World Bank) Haeil Jung (Korea University) Angela Kinnell (University of Adelaide) Amelia Maika (Gadjah Mada University) Nozomi Nakajima (Harvard University) Menno Pradhan (University of Amsterdam and VU University)

# **JEL Codes**

I24, I25

# **Keywords**

Early Childhood Education, Child Development, Treatment impacts, Differential effects

*Authors are listed alphabetically.
# **Acknowledgments**

We would like to thank Dedy Junaedi, Upik Sabainingrum, Anas Sutisna, Lulus Kusbudiharjo and Mulyana for managing the fieldwork. We are grateful to Sophie Naudeau, Alaka Holla and Halsey Rogers for comments on an earlier version. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.

# **Funding**

This work was supported by the Strategic Impact Evaluation Fund (TF0A0234) at the World Bank <mark>. Haeil Jung acknowledges that this work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF2016S1A3A2924956).</mark>

# **Declarations of interest**

None.
# **1. Introduction**

A growing body of research shows that a child’s early life has consequences for later life outcomes in education (Bhutta et al., 2002; Brennan et al., 2012; Duncan et al., 2007; Feinstein and Duckworth, 2006; Melhuish et al., 2008; Moser et al., 2012), health (Hertzman, 2013), and social capital (Moffitt et al., 2011). Healthy child development is an enabler of human capability allowing children to reach physical maturity and participate productively in economic, social and civic life (Conti and Heckman, 2012; Sen, 1999). Many of the problems arising in early childhood have social and financial costs that cumulatively represent a considerable drain on a country’s resources (Feinstein and Duckworth, 2006; Victora et al., 2008).

High-quality preschool programs provide an opportunity to mitigate the risk factors that many young children face (Barnett, 2011; Duncan et al., 2007; Duncan and Magnuson, 2013; Heckman, 2006; Shonkoff and Phillips, 2000). Given the high-risk factors faced by children growing up in middle- and low-income countries (Engle et al., 2011), the effectiveness of preschool programs is likely to be large for these children. However, much of the evidence base on the long-term impacts of preschools has focused on three “iconic” projects in the United States: Perry Preschool, Abecedarian, and the Nurse Family Partnerships (Shonkoff, 2014). These studies examined high-intensity interventions in the United States that were run with small sample sizes in the late 1960s to early 1970s. While all have had longitudinal followups, few interventions have been implemented since that match either the fidelity or the intensity of these interventions.

In developing country settings, rigorous evaluations of preschool programs have emerged in the last decade. Results have ranged from no effect in some settings (Bouguen, Filmer, Macours and Naudeau, 2017) to positive effects in others (Brinkman et al., 2017; Jung and Hasan, 2016; Burger, 2010; Martinez, Naudeau and Pereira, 2012; Nores and Barnett, 2010). Systematic reviews of preschool interventions in developing countries reinforce the wide range of impacts seen in international settings (Tanner et al., 2015). The contrasting evidence from different settings has led some to question whether early childhood education can even have lasting impacts (Stevens and English, 2016).

In this paper, we contribute to the growing literature on the longer-term impacts of preschools in developing countries. We examine the longer-term impacts of early childhood education in a developing country setting using data on two cohorts of children from the Indonesia Early Childhood Education and Development (ECED) Project. The project lasted five years and expanded access to community-based preschools by providing block grants to villages to establish up to two playgroups (targeted for ages 3-4), providing teacher training, and raising community awareness about the importance of early education.<sup>1</sup> The _older cohort_ in our study was age 4 and the _younger cohort_ was age 1 when the project began.

The older and younger cohorts differed in terms of (1) the phase of project implementation when they were exposed to playgroups and (2) the length of exposure to playgroups at the appropriate age. On the first distinction, the older cohort was exposed to playgroups during the project’s first year since they were already age 4 at the start of the project. In contrast, the younger cohort was exposed to playgroups during the project’s third year when they turned 3 years old. As a result, the two cohorts experienced the same project at different phases of implementation. On the second distinction, the older cohort was at the upper-end of the appropriate age for playgroup attendance (age 3-4) when they were exposed to the project. In contrast, the younger cohort was only 1 year old when the project started, which meant that

> 1 In practice, at the start of the project, the playgroups allowed enrollment for children ages 3-6. See Brinkman et al, 2017a for further discussion.

1
they had an opportunity to enroll in playgroups at the appropriate age, starting at age 3 and continuing through age 4. Thus, the two cohorts differed in their likelihood of being exposed to playgroups at the appropriate age.

Our data allow us to track the development outcomes for both cohorts at ages 5 and 8. To control for children’s baseline development, we use measures of child development before exposure to playgroups. For the younger cohort, we have data on child development measures from when they were 1 and 2 years old. For the older cohort, we have data on child development measures from when they were 4 years old. We use a comprehensive set of child development outcomes that measure both cognitive and socio-emotional development. In addition, we capture children’s performance on a test of language, mathematics and abstract reasoning in primary school. Together, these measures allow us to trace out early development on a variety of dimensions.

We build upon previous work by Brinkman et al. (2017b), which analyzed the impact of the project on the older cohort. In this paper, we conduct three new analyses. First, we present child development outcomes for the younger cohort at ages 5 and 8. Second, we present new results for the older cohort using primary school test scores. Third, we contrast the developmental trajectories between the two cohorts. We explore how features of project implementation as well as different rates of exposure to playgroups at the appropriate age may have contributed to the contrasting results between the two cohorts.

This paper also contributes to research on the generalizability of impact evaluations in early childhood education. Preschool interventions in developing countries have produced a wide range of effect sizes, prompting both researchers and policy makers to question the generalizability of these findings (Tanner et al., 2015). In particular, it is difficult to understand what causes these variations since each preschool program is different in terms of what the program provides, how the program is implemented and where the program is carried out. In this paper, we are able to hold constant the program content (what) and the study context (where) but vary the implementation (how) experienced by two cohorts of children. In doing so, we document the extent to which program impacts may vary as a result of differences in project implementation.

This paper proceeds as follows: The second section describes the Indonesia Early Childhood Education and Development (ECED) Project. The third section explains the data we use in our analysis. The empirical strategy is described in the fourth section. The fifth section provides the empirical results and the sixth section compares the impacts on the younger cohort and older cohorts. The seventh section presents a cost-benefit analysis based on our intent-to-treat estimates of additional years of schooling completed by those living in project villages. The final section concludes with a discussion of the findings and their implications for future work.

# **2. Indonesia Early Childhood Education and Development (ECED) Project**

The goals of the Indonesia ECED Project were to increase access to early childhood services and to increase children’s school readiness in rural villages. The project consisted of three components. First, a community facilitator raised awareness about the importance of early childhood services and shared information about how to prepare a proposal for the block grants available through the project.

Second, block grants were provided to each village, in the amount of USD 18,000 per village over three years. Villages could use the grant to establish or support up to two early childhood education centers. No more than 20 percent of the grant could be spent on new infrastructure. The most common form of services established were playgroups, which are early childhood education services catered to children ages 3 and 4 before they enroll in 2
kindergarten at ages 5 and 6. Playgroups are typically half-day programs that meet every other day while kindergartens typically meet daily.

Third, the project included a component that provided 200 hours of teacher training for up to two teachers per project playgroup. Teachers were predominantly mothers in the villages, some of whom had prior work experience in health and education services.

Thus, the “treatment” evaluated in this impact evaluation refers to this package of interventions provided by the ECED project – a community facilitator, block grants to establish playgroups, and teacher training (Hasan, Hyson and Chang, 2013).

The project was implemented in relatively poor villages in rural Indonesia. Of the 442 districts in the country, 50 poor districts were selected based on having high poverty rates, low enrollment rates in early childhood education, and low Human Development Index rankings. Within each district, 60 priority villages were identified based on their poverty rate, population size, and willingness to participate in the project. Overall, the Indonesia ECED Project was implemented in 3,000 villages.<sup>2</sup> According to project documents, the cost per child for the project’s package of interventions was about US$30 per year. This estimate excludes any voluntary contributions from the villages to the project. Villages often made available the land on which playgroups were housed. In contrast, other early childhood programs range in cost from US$37 per child in India to US$52 per child in Mexico to US$66 per child in Brazil—suggesting that this package was slightly less costly.<sup>3</sup>

# _Evaluation design_

The impact evaluation of the project focuses on 310 villages in 9 districts that were part of the project. There are 218 treatment villages and 92 comparison villages. The treatment villages were randomly assigned to two batches. 105 villages received the project first in 2009 (referred to as _batch 1_ ) and 113 villages received the project 11 months later in 2010 (referred to as _batch 2_ ). The comparison villages never received the project. The district governments selected the comparison villages on the basis of having similar poverty levels to the treatment villages.<sup>4</sup> Comparison villages were therefore not randomly assigned.

In each treatment and comparison village, approximately 10 households with a 1 year-old child (who became the younger cohort) and approximately 10 households with a 4 year-old child (who became the older cohort) were randomly selected for evaluation.<sup>5</sup> Thus, the impact evaluation follows these two separate cohorts of children who were able to access the playgroups provided by the project at different time points, based on their age, and when the project was at different stages of maturity. The timeline in Figure 1 below shows the timing of the project and the ages of the two cohorts.

An earlier paper by Brinkman et al. (2017b) documented the impacts of the project on the older cohort which, based on their age, was eligible to enroll in playgroups established under the project as soon as the project was implemented in 2009 and 2010. It employed instrumental variables and difference-in-differences models to determine the impacts on these children when they were aged 5 and 8 respectively. The paper found that while the intervention raised enrollment rates and durations of enrollment, there was little impact on child development. The two models corresponded to different durations of project exposure. The

> 2 For further details please see Pradhan et al. (2013).

> 3 See for instance the estimates quoted in Barnett (1997) and Evans, Myers and Ilfeld (2000).

> 4 Appendix 7-9 in Hasan, Hyson and Chang (2013) document that these villages are well balanced on a range of observable characteristics.

> 5 Only 32 children from the two cohorts are siblings to each other.

3
difference-in-differences model captured greater exposure and showed that there were modest and sustained impacts on child development, especially for children from more disadvantaged backgrounds.

The present paper focuses on the younger cohort who, based on their age, became eligible to enroll in playgroups later in the project’s implementation, in 2011 and 2012. Similar to the previous analysis for the older cohort (Brinkman et al. 2017b), this paper estimates the impact of the intervention for the younger cohort at age 5 and at age 8 in terms of enrollment in pre-primary education and child development outcomes. We also capitalize on the two cohorts that experienced the project to contrast the impact estimates for the younger cohort with those of the older cohort. For the older cohort, in addition to estimates presented in Brinkman et al. (2017b), we report new estimates of the impact of exposure to early childhood education and development services on test scores in language (Bahasa Indonesia) conducted when the children were aged 8 and in the early grades of primary school.

4
Figure 1: Age of cohort, eligibility for various early childhood education, and project implementation phase for each survey round

|**Year**|**2009**|**2010**|**2011**|**2012**|**2013**|**2014**|**2015**|**2016**|
|---|---|---|---|---|---|---|---|---|
|**Survey**|**Round 1**|**Round 2**|||**Round 3**|||**Round 4**|
|**Project phase**|**Start of**<br>**project**||||**End of**<br>**project**||||
|**1-year-old**<br>**cohort**|**1**|**2**|3|4|**5**|6|7|**8**|
|Type of service<br>child is eligible<br>to enroll in|Children are t<br>enroll in pl|oo young to<br>aygroups|Children ar<br>enroll in p|e eligible to<br>laygroups|Children ar<br>enroll in ki|e eligible to<br>ndergartens|Children<br>enroll in p|are eligible to<br>rimary school|
|**4-year-old**<br>**cohort**|**4**|**5**|6|7|**8**|9|10|11|
|Type of service<br>child is eligible<br>to enroll in|Children are<br>eligible to<br>enroll in<br>playgroups|Children are<br>enroll in kin|eligible to<br>dergartens|Children a<br>enroll in pr|re eligible to<br>imary school|_These childr_<br>_but would h_|_en were no lo_<br>_ave been in p_|_nger surveyed_<br>_rimary school_|

Note: Figure depicts ages of the two cohorts studied and what types of services they are eligible for at each age. Note: Bold numbers denote that the children in that cohort were surveyed at that age/year.

Source: Authors 5
# **3. Data**

The main analyses in this paper use data on the younger cohort collected in 2013 (at age 5) and in 2016 (at age 8). The key outcomes of interest are (i) enrollment in different types of early childhood education services; (ii) child development outcomes using the Early Development Instrument (EDI); and (iii) tests scores in early grades of primary school.

We measure enrollment in three types of early childhood education services. The first are project playgroups, those established under the project’s block grant. The second are nonproject playgroups, which refer to all other playgroup services that exist in the communities. The third are kindergartens, which are early childhood education programs catering to children before they enroll in primary school. We collected information about enrollment in each type of service by collecting a retrospective enrollment history for each academic year from 2008 to the survey year from the primary caregiver of the child.

The EDI measures children’s school readiness across five domains: physical health and well-being, social competence, emotional maturity, language and cognitive development, and communication skills and general knowledge (Janus and Offord, 2007). The EDI has been validated and tested for reliability. Overall, the construct validity, predictive validity, and interrater reliability of the EDI in Indonesia are comparable to that found in other countries, making the EDI a suitable instrument for measuring school readiness in Indonesia (Brinkman et al. 2016). In this paper, we use the short-form of the caregiver-rated EDI.<sup>6</sup> We standardized the variables for each EDI domain to have a mean of 0 and standard deviation of 1, using the mean and standard deviation of the comparison group in the younger cohort.

A school-based test was developed for this evaluation based on learning standards in Indonesian schools. Children were assessed in a classroom under the guidance of a member of the data collection team and no classroom teachers were present. The tests were divided into three parts: language (Bahasa Indonesia), mathematics, and abstract reasoning. <mark>The language test consisted of two sections. The first section (</mark> _<mark>match pictures</mark>_ <mark>) evaluated children’s phonological awareness (i.e., whether they can match pictures that start with a given sound) and letter recognition (i.e., whether they can match pictures that start with a given letter). The second section (</mark> _<mark>mention objects</mark>_ <mark>) assessed children’s vocabulary skills (i.e., whether they can name the word associated with a given image). The mathematics test included two sections. The first section (</mark> _<mark>summation</mark>_ <mark>) evaluated children’s ability to add and subtract (i.e., whether they can add to or subtract away from a set of objects). The second section (</mark> _<mark>order numbers</mark>_ <mark>) assessed children’s ability to recognize patterns (i.e., whether they can order one- to two- digit numbers in ascending and descending order). The abstract reasoning section was</mark> modeled on the Raven’s Progressive Matrices. Students were presented with an image that was missing a small section and asked to select the missing pieces from six options, based on color, pattern, and orientation. There were two versions of the overall test; a shorter test for 6 and 7 year-olds and a longer test for 8 and 9 year-olds. In this paper, we use the common set of items that were included in both versions of the test. We standardized the variables for each test domain to have a mean of 0 and standard deviation of 1, using the mean and standard deviation of the comparison group in the younger cohort.

Baseline measures of child development were collected for the younger cohort when they were aged 1 and 2. The EDI is not an appropriate test for that age group so instead a measure of child development developed by the University of San Carlos Office of Population

> 6 We use the short-form to match previously published estimates in Brinkman et al 2017b with which we compare this paper’s findings. Results using the long-form of the EDI are qualitatively similar and are available upon request.

6
Studies that measured skills similar to those in the EDI was used. These measures were collected by asking the child’s primary caregiver whether the child is usually able to demonstrate various skills. Specifically, we directly observed (or, with younger or reluctant children, asked the mother about) children’s gross and fine motor skills, language, cognitive and socio-emotional development. In one set of questions, children were asked to demonstrate their ability to perform a specified skill. When the child did not want to demonstrate this skill, the mother was asked if the child was usually able to do it. In another set of questions, the mother was asked directly whether their child could perform a particular activity. For these skills, the child was never asked to do a demonstration. In all cases, higher values indicate better developmental outcomes (Office of Population Studies, 2005). We standardized each of the variables to have a mean of 0 and standard deviation of 1, using the mean and standard deviation of the comparison group in the younger cohort.

Although all children in the sample are from poor, rural areas, we measured the relative wealth of children’s households. Households were asked if they owned any of the following items: radio, television, refrigerator, bicycle, motorcycle, car, mobile phone, and livestock. They were also asked about the materials used to construct the floor, walls and roof of their homes. Households were also asked if they had access to electricity in their homes and whether they received government assistance. Using principal components analysis on these items, we constructed a single index of household wealth. We standardized the variable to have a mean of 0 and standard deviation of 1, using the younger cohort’s mean and standard deviation.<sup>7</sup> As a measure of the child-parent relationship, we collected self-reports from the primary caregiver on how often they used various parenting practices related to their warmth, consistency, and hostility. The questionnaire was adapted from the Longitudinal Study of Australian Children (Zubrick et al. 2008). Higher total parenting scores correspond to higher levels of warmth and consistency, and lower levels of hostility. We standardized the variable to have a mean of 0 and standard deviation of 1, using the younger cohort’s mean and standard deviation.

# **4. Empirical Strategy**

To estimate the causal effect of the project on the younger cohort, we would ideally compare the change in outcomes between age 1 and age 5 (or between age 1 and age 8) for children in the treatment villages, relative to the change in outcomes for children in the comparison villages (i.e., a difference-in-differences approach). However, children in the younger cohort were too young to have baseline measures of enrollment in early childhood education as they were not yet age eligible. These children were also too young to have baseline measures of the EDI as they were not old enough for the instrument.

Instead, we evaluate the impact of the project using the following regression specification:

> 7 A comparison of assets ownership by households in the evaluation sample with that of the rural subsample of the SUSENAS (a nationally representative household survey) suggests average rates of asset ownership and education levels are by and large similar between the two. See also Hasan, Hyson and Chang (eds.) 2013.

7
where y��� is the outcome measure of child _i_ in village _j_ at time _t._ T� is an indicator for whether the village is treatment or comparison, X��� are time varying covariates (child’s age, household size, household wealth index, and parenting score) and V�������� are time invariant covariates (child’s gender, whether the child’s mother completed primary education or less, and baseline measures of child development). The key coefficient of interest is the treatment effect, α� . Equation (1) is the specification for 2013, which examines the effect of the intervention on enrollment rates and EDI at age 5. Equation (2) estimates the impact in 2016, which examines the effect of the project on enrollment rates, EDI, and test scores at age 8.

Our key identifying assumption is that the time varying and invariant covariates in our regression model fully account for any differences between children in the treatment villages and children in the comparison villages that are not due to treatment assignment.

We also examine the heterogeneity of the treatment effect across household wealth and parenting practices. We re-run our regression model separately for children with baseline household wealth below the sample mean ( _poor_ ) and for children with baseline household wealth above the sample mean ( _non-poor_ ). Similarly, we re-run our regression model separately for children with baseline parenting scores below ( _low parenting score_ ) and above ( _high parenting score_ ) the sample mean.

In Table 1, for the younger cohort, we show the summary statistics of child and family characteristics in treatment and comparison villages at baseline (2009). Columns 1 and 2 separately present the means and standard deviation for villages that received the project early (batch 1) and those that received the project later (batch 2), and column 3 presents these statistics for a sample that combines all treated villages together. Column 4 reports these statistics for the comparison villages that never received the project but were chosen because of their similarity to treatment villages. Column 5 reports the differences between villages that received the project early or late while column 6 reports the differences between treatment and comparison villages. In both of these cases the estimates reported are the results of a regression with standard errors clustered at the village level.

Column 3 shows that at baseline, younger cohort children in the treatment villages were around 1.5 years old. On average, children lived in households with wealth z-scores and parenting z-scores slightly below the sample mean. About half of the cohort’s mothers had primary education or less and about half of the children were male. The mean body mass index (BMI) of the children in the younger cohort was 14.6 kg/m<sup>2</sup> and on a range of cognitive, fine motor, gross motor and language skills their scores were slightly below the sample mean.

Column 5 reports that there are no differences in these child and family characteristics between the two batches of treated villages. As a result, we examine batch 1 and batch 2 villages collectively as treatment villages in our regression specification in equations (1) and (2).

Column 6 shows that at baseline, treatment and comparison villages were generally similar in terms of various child and family characteristics. However, three variables showed statistically significant differences. On average, children in treatment villages lived in households with 0.259 fewer people than children in comparison villages. While this mean difference is statistically significant, the magnitude is small. We also find that children in treatment villages scored lower in measures of baseline cognitive skills (-0.143 S.D.) and gross motor skills (-0.153 S.D.). Thus, we control for these baseline differences in child development in our estimation of the treatment effect.

8
<u>Table 1. Summary statistics for the younger cohort at baseline (2009)</u>

|||<br>**Treatment**|h|**Ci**|**D**<br>l|**ifferences**|
|---|---|---|---|---|---|---|
||Early (Batch 1)|Late (Batch 2)|Bot<br>(Batch 1 & 2)|**omparson**|Eary –<br>Late|Both – Comparison|
||(1)|(2)|(3)|(4)|(5)|(6)|
|A ()|1.520|1.499|1.509|1.508|-0.021|0.001|
|ge years|(0.287)|(0.286)|(0.286)|(0.286)|(0.012)|(0.013)|
|Household size|4.678|4.729|4.704|4.964|0.052|-0.259*|
||(1.529)|(1.568)|(1.549)|(1.705)|(0.095)|(0.102)|
|Wealth z-score (SD)|-0.0125|-0.0301|-0.0217|0.0520|-0.018|-0.074|
|..|(1.036)|(0.962)|(0.998)|(1.003)|(0.082)|(0.073)|
|Pti SD|-0.0120|-0.0316|-0.0222|0.0532|-0.020|-0.075|
|arenng z-score (..)|(1.006)|(0.968)|(0.986)|(1.031)|(0.070)|(0.073)|
|Mth’ d i i l 1 = Y|0.512|0.514|0.513|0.504|0.002|0.008|
|oers eu s prmary or ess ( es)|(0.500)|(0.500)|(0.500)|(0.500)|(0.029)|(0.030)|
|Ml 1 = Y|0.491|0.516|0.504|0.525|0.025|-0.021|
|ae ( es)|(0.500)|(0.500)|(0.500)|(0.500)|(0.023)|(0.019)|
|Bod Mass Index (BMI)|14.62|14.53|14.57|14.66|-0.092|-0.091|
|y|(2.078)|(2.117)|(2.098)|(2.097)|(0.107)|(0.102)|
|Citi kill SD|-0.0979|0.00907|-0.0421|0.101|0.107|-0.143*|
|ognve ss (..)|(1.055)|(0.982)|(1.019)|(0.947)|(0.076)|(0.061)|
|Fi kill SD|-0.0323|-0.00911|-0.0202|0.0483|0.023|-0.069|
|ne motor ss (..)|(0.988)|(1.045)|(1.018)|(0.954)|(0.059)|(0.055)|
|G kill SD|-0.0798|-0.0134|-0.0452|0.108|0.066|-0.153**|
|ross motor ss (..)|(1.013)|(1.017)|(1.015)|(0.954)|(0.057)|(0.047)|
|L kill SD|-0.0216|-0.00990|-0.0155|0.0371|0.012|-0.053|
|anguage ss (..)|(0.984)|(1.031)|(1.008)|(0.979)|(0.058)|(0.052)|
|Observations|1042|1137|2179|910|||

p<0.001***; p<0.01**; p<0.05* Note: Early (Batch 1) villages received the project first in 2009 and late (batch 2) villages received the project later in 2010. Comparison villages never received the project. Standard deviation in parentheses in columns (1) to (4). Standard errors clustered at village level in parentheses in columns (5) to (6).

9
# **5. Results**

The intent-to-treat impact estimates for the younger cohort are presented in Tables 2 to 4.<sup>8</sup> In each table, column 1 presents the estimates for all children in the cohort, columns 2 and 3 separately estimate the impacts by relative household wealth at baseline, and columns 4 and 5 separately estimate the impacts by relative parenting score at baseline.<sup>9</sup> When interpreting these results, it is important to note that the counterfactual is children living in comparison villages, which may or may not have other preschool services (i.e., non-project playgroups and kindergartens) available in their villages.

Table 2 presents impacts on enrollment rates and duration. In 2013 (at age 5), children in treatment villages were 49.9 percentage points more likely to report ever being enrolled in project playgroups compared to children from comparison villages. The treatment effect on enrollment rate was similar in 2016 when the children were age 8. Moreover, the effects are largely consistent across sub-samples.<sup>10</sup> As expected, there was virtually no enrollment in project playgroups reported by children in comparison villages. In contrast to the increase in enrollment in project playgroups, children from treatment villages were 22 percentage points less likely to enroll in non-project playgroups relative to a 33.4 percent enrollment rate among children in comparison group villages. These estimates were fairly similar at ages 5 and 8. Finally, there was no difference in enrollment in kindergarten between treatment and comparison villages by age 5. However, by age 8, children from treatment villages were 8.6 percentage points less likely to have ever enrolled in kindergartens compared to children from comparison villages.

The results for months of enrollment are largely consistent with our findings for enrollment rate. The project increased children’s enrollment duration in project playgroups, decreased enrollment duration in non-project playgroups, and kept enrollment duration in kindergarten unaffected.

One way to interpret the months of enrollment in a project playgroup is that it is a measure of “take-up” of the project. The average take-up of the project playgroup in treatment villages was 7.781 months by age 5 and 8.683 months by age 8.<sup>11</sup> These large effects on months of enrollment in project playgroups hold across wealth and parenting sub-group analyses. Specifically, compared to poor children in comparison villages, poor children in project villages enrolled in 7.3 more months of playgroup. Non-poor children enrolled for 8.1 more months. However, there is no evidence to suggest that these point estimates are different from each other.<sup>12</sup> The project thus had equally large enrollment effects for children from poor and non-poor households. Similarly, we do not find treatment effect variation between children from households with high and low parenting scores.

On average, children in treatment villages were enrolled in non-project playgroups for 3 fewer months than their peers in comparison villages. At age 5, this decrease in enrollment duration in non-project playgroups was significantly more pronounced for children from nonpoor households. However, by age 8, there was no treatment effect variation in non-project

> 8 Results by batch are reported in Appendix Tables 1-3.

> 9 In both cases – wealth and parenting practices – we split the sample into those above the mean and those below the mean.

> 10 Appendix Table 4 provides results which show that the differences between groups – either poor versus non-poor or low versus high parenting practices are not statistically different from each other.

> 11  Since all of our impact estimates focus on intent-to-treat, these take-up figures do not affect the interpretation of our results. The small non-zero estimates of enrollment in project centers among children from non-project villages are possible in those few cases where households lived near enough to a treatment community and project playgroup. However, this was rare.

> 12 See Appendix Table 4.

10
playgroup enrollment between poor and non-poor children.

Finally, we do not find significant differences in enrollment duration in kindergarten between treatment and comparison villages, either in 2013 or in 2016. On average, all children in the sample seem to have completed about 5 months of kindergarten by 2013 and about 10 months of kindergarten by 2016 – with no substantial variation by household wealth or parenting.

<u>Table 2. Impact on enrollment outcomes for the younger cohort</u>

|**Outcome**|**Survey**<br>**Year**||**All**|**Poor**|**Non-poor**|**Low**<br>**Parenting**<br>**Score**|**High**<br>**Parenting**<br>**Score**|
|---|---|---|---|---|---|---|---|
||||(1)|(2)|(3)|(4)|(5)|
|||Coeff.|0.499***|0.494***|0.504***|0.501***|0.498***|
||2013|(S.E.)|(0.022)|(0.029)|(0.024)|(0.025)|(0.026)|
|||Comp. mean|0.00732|0.0116|0.00421|0.00699|0.00767|
|Ever enrolled in<br>l||Obs.|2,778|1,201|1,577|1,540|1,238|
|project paygroup<br>until surve ear||Coeff.|0.498***|0.539***|0.470***|0.466***|0.532***|
|y y|2016|(S.E.)|(0.023)|(0.029)|(0.027)|(0.027)|(0.026)|
|||Comp. mean|0.0673|0.0499|0.0802|0.0686|0.0659|
|||Obs.|2,894|1,289|1,605|1,834|1,060|
|||Coeff.|-0.220***|-0.193***|-0.241***|-0.215***|-0.223***|
||2013|(S.E.)|(0.037)|(0.044)|(0.042)|(0.041)|(0.043)|
|Ever enrolled in||Comp. mean|0.334|0.284|0.371|0.324|0.345|
|non-project||Obs.|2,778|1,201|1,577|1,540|1,238|
|playgroup until||Coeff.|-0.221***|-0.190***|-0.241***|-0.209***|-0.232***|
|survey year|2016|(S.E.)|(0.037)|(0.045)|(0.042)|(0.041)|(0.045)|
|||Comp. mean|0.407|0.338|0.459|0.389|0.427|
|||Obs.|2,894|1,289|1,605|1,834|1,060|
|||Coeff.|-0.075|-0.074|-0.085*|-0.090|-0.059|
||2013|(S.E.)|(0.040)|(0.053)|(0.041)|(0.048)|(0.043)|
|||C|0532|0441|0598|0522|0542|
|E lld i||omp. mean|.|.|.|.|.|
|ver enroe n<br>kindergarten until||Obs.|2,778|1,201|1,577|1,540|1,238|
|<br>survey year||Coeff.|-0.086*|-0.068|-0.098*|-0.098*|-0.075|
||2016|(S.E.)|(0.040)|(0.054)|(0.038)|(0.044)|(0.046)|
|||Comp. mean|0.743|0.670|0.796|0.735|0.751|
|||Obs.|2,894|1,289|1,605|1,834|1,060|
|||Coeff.|7.781***|7.319***|8.129***|7.748***|7.843***|
||2013|(S.E.)|(0.396)|(0.511)|(0.447)|(0.454)|(0.469)|
|||Comp. mean|0.0622|0.122|0.0189|0.0559|0.0691|
|Months enrolled in<br>||Obs.|2,778|1,201|1,577|1,540|1,238|
|project playgroup<br>til||Coeff.|8.683***|8.840***|8.599***|8.365***|9.022***|
|un survey year||(S.E.)|(0.413)|(0.544)|(0.469)|(0.496)|(0.491)|
||2016|<br>Comp. mean|<br>0.327|<br>0.321|<br>0.331|<br>0.320|<br>0.334|
|||<br>Obs.|2,894|1,289|1,605|1,834|1,060|
|||Coeff.|-3.001***|-2.094***|-3.743***|-2.721***|-3.284***|
|||(S.E.)|(0.637)|(0.571)|(0.844)|(0.629)|(0.798)|
|Months enrolled in|2013|Comp. mean|4.635|3.290|5.613|4.228|5.082|
|non-project||<br>Obs.|2,778|1,201|1,577|1,540|1,238|
|<br>playgroup until||Coeff.|-3.187***|-2.476***|-3.706***|-2.913***|-3.443***|
|survey year|2016|(S.E.)|-0.64|(0.651)|(0.814)|(0.660)|(0.796)|
|||Comp. mean|5.346|4.241|6.167|4.975|5.741|
|||Obs.|2,894|1,289|1,605|1,834|1,060|

11
<u>Table 2. Impact on enrollment outcomes for the younger cohort</u>

|<br>**Outcome**|<br>**Survey**<br>**Year**|<br> <br>|<br>**All**<br>(1)|<br>**Poor**<br>(2)|**Non-poor**<br>(3)|**Low**<br>**Parenting**<br>**Score**<br>(4)|**High**<br>**Parenting**<br>**Score**<br>(5)|
|---|---|---|---|---|---|---|---|
|||Coeff.|-0.199|0.068|-0.516|-0.151|-0.273|
||2013|(S.E.)|(0.436)|(0.505)|(0.496)|(0.479)|(0.513)|
|||Comp. mean|4.983|3.777|5.859|4.555|5.453|
|Months enrolled in<br>||Obs.|2,778|1,201|1,577|1,540|1,238|
|kindergarten until<br>re ear||Coeff.|-1.012|-0.589|-1.326|-1.365|-0.692|
|suvy y|2016|(S.E.)|(0.663)|(0.844)|(0.683)|(0.722)|(0.767)|
|||Comp. mean|10.24|9.008|11.16|10.06|10.43|
|||Obs.|2,894|1,289|1,605|1,834|1,060|

p<0.001***; p<0.01**; p<0.05* Note: Each cell-block is the result of a separate regression. Standard errors clustered at the village level are in parentheses. “Comp. mean” refers to the comparison group mean for the outcome variable. Column (1) regressions control for child characteristics (age, household size, household wealth asset index, parenting practices, mother's education and gender) and baseline child development measures (BMI, cognitive, fine motor, gross motor, and language). Columns (2) and (3) regressions use the same controls as column (1) except they exclude household wealth. Columns (4) and (5) regressions use the same controls as column (1) except they exclude parenting practices.

Next, we turn to results on child development outcomes. Table 3 presents the impact estimates on the EDI at age 5 (2013) and age 8 (2016). Overall, we find a few positive impacts of the project on children’s developmental outcomes at age 5 but no positive impacts at age 8. At age 5, we estimate a 0.208 S.D. increase in scores on the physical health and well-being domain and a 0.115 S.D. increase in scores on the emotional maturity domain for the overall sample. However, these effects do not persist to age 8. We generally observe null effects in 2016, with one negative impact on the communication skills and general knowledge domain (- 0.136 S.D). These results are largely consistent with the fade-out literature in early childhood education, where interventions aimed to reduce early childhood inequalities have large initial impacts but these dissipate over time (see for instance the studies included in Tanner et al., 2015). Our results suggest that the impact on emotional maturity may be concentrated among those classified as poor in our data.<sup>13</sup> There are no other statistical differences across subgroups – either by wealth or by parenting practices.

One note of caution in interpreting our results is warranted. The data suggest that measurement error may contribute to the null effects on the EDI that we observe in 2016. In the raw histograms of the EDI domains presented in Appendix Figure 1, we find evidence of ceiling effects in 2016 that are not present in 2013. The exception in 2013 was ceiling effects on the communication skills and general knowledge domain, which may also explain the negative impact on this domain identified in 2016.

13 See Appendix Table 5.

12
Table 3. Impact on EDI outcomes for the younger cohort

|**Outcome**|**Survey**<br>**Year**||**All**|**Poor**|**Non-poor**|**Low**<br>**Parenting**<br>**Score**|**High**<br>**Parenting**<br>**Score**|
|---|---|---|---|---|---|---|---|
|||Coeff.|0.208***|0.115|0.267***|0.151*|0.275***|
||2013|(S.E.)|(0.051)|(0.082)|(0.059)|(0.064)|(0.072)|
|||Comp. mean|-0.149|-0.118|-0.172|-0.191|-0.102|
|Physical health and||Obs.|2,770|1,194|1,576|1,533|1,237|
|well-being (SD)||Coeff.|0.023|0.113|-0.047|0.013|0.025|
||2016|(S.E.)|(0.065)|(0.103)|(0.060)|(0.085)|(0.069)|
|||Comp. mean|0.00610|-0.170|0.136|-0.153|0.174|
|||Obs.|2,877|1,279|1,598|1,823|1,054|
|||Coeff.|0.018|0.080|-0.031|0.014|0.016|
||21|(S.E.)|(0.053)|(0.082)|(0.058)|(0.056)|(0.079)|
||03|Comp. mean|0.00112|-0.192|0.140|-0.139|0.157|
|il||Obs.|2,769|1,192|1,577|1,534|1,235|
|Soca competence (SD)||Coeff.|0.008|0.052|-0.023|-0.036|0.040|
||2016|(S.E.)|(0.049)|(0.067)|(0.058)|(0.070)|(0.059)|
|||Comp. mean|0.0284|-0.0539|0.0892|-0.249|0.321|
|||Obs.|2,877|1,279|1,598|1,823|1,054|
|||Coeff.|0.115*|0.241**|0.025|0.115|0.114|
||2013|(S.E.)|(0.056)|(0.082)|(0.057)|(0.076)|(0.059)|
|||Comp. mean|-0.110|-0.228|-0.0251|-0.333|0.138|
|Etil tit SD||Obs.|2,770|1,193|1,577|1,534|1,236|
|moona maury ()||Coeff.|0.020|0.098|-0.042|-0.021|0.049|
|||(S.E.)|(0.046)|(0.061)|(0.058)|(0.064)|(0.059)|
||2016|Comp. mean|-0.00663|-0.145|0.0955|-0.305|0.307|
|||Obs.|2,877|1,279|1,598|1,823|1,054|
|||Coeff.|0.073|0.098|0.043|0.098|0.051|
||2013|(S.E.)|(0.056)|(0.084)|(0.061)|(0.064)|(0.074)|
|||Comp. mean|-0.0502|-0.285|0.117|-0.154|0.0648|
|Language and cognitive||Obs.|2,770|1,193|1,577|1,534|1,236|
|development (SD)||Coeff.|0.080|0.078|0.090|0.085|0.067|
||21|(S.E.)|(0.049)|(0.074)|(0.056)|(0.067)|(0.067)|
||06|Comp. mean|-0.0371|-0.110|0.0169|-0.103|0.0324|
|||Obs.|2,877|1,279|1,598|1,823|1,054|
|||Coeff.|-0.005|-0.095|0.055|-0.069|0.067|
|||(S.E.)|(0.074)|(0.101)|(0.076)|(0.081)|(0.091)|
||2013|<br>Comp. mean|<br>0.0159|<br>-0.0462|<br>0.0603|<br>0.00577|<br>0.0271|
|Communication skills<br>d l kld||<br>Obs.|2,771|1,194|1,577|1,534|1,237|
|an genera nowege<br>||Coeff.|-0.136*|-0.145*|-0.123|-0.153|-0.127|
|(SD)||(S.E.)|(0.061)|(0.072)|(0.074)|(0.078)|(0.067)|
||2016|Comp. mean|0.131|0.137|0.127|-0.0184|0.289|
|||Obs.|2,877|1,279|1,598|1,823|1,054|

p<0.001***; p<0.01**; p<0.05* Note: Each cell-block is the result of a separate regression. Standard errors clustered at the village level are in parentheses. “Comp. mean” refers to the comparison group mean for the outcome variable. Column (1) regressions control for child characteristics (age, household size, household wealth asset index, parenting practices, mother's education and gender) and baseline child development measures (BMI, cognitive, fine motor, gross motor, and language). Columns (2) and (3) regressions use the same controls as column (1) except they exclude household wealth. Columns (4) and (5) regressions use the same controls as column (1) except they exclude parenting practices.

In Table 4, we find mixed results of the project on primary school test scores, which were measured at age 8 (2016). We find large positive effects (0.134 S.D.) on language items involved with selecting a picture whose name began with a different letter to other pictures 13
( _match picture_ ) but null effects on language tasks associated with writing the name of everyday items ( _mention objects_ ). For the mathematics section, we similarly find large positive effects (0.125 S.D.) on tasks associated with ordering sequences of numbers from largest to smallest and vice versa ( _order numbers_ ) but null effects on solving addition problems ( _summation_ ) _._ Finally, we find no impact on abstract reasoning. Overall, we do not detect treatment effect variation in the test score results, either by wealth or by parenting practices.<sup>14</sup> As was the case for the EDI outcomes, test scores also seem to have ceiling effects associated with them, which may undermine our ability to detect certain effects.<sup>15</sup> <u>Table 4. Impact on primary school test scores for the younger cohort</u>

|**Outcome**||**All**|**Poor**|**Non-poor**|**Low**<br>**Parentin**<br>**g Score**|**High**<br>**Parenting**<br>**Score**|
|---|---|---|---|---|---|---|
||Coeff.|0.134*|0.029|0.213**|0.179**|0.080|
|Language - match picture|(S.E.)|(0.058)|(0.081)|(0.068)|(0.067)|(0.077)|
|(SD)|Comp. mean|-0.0827|-0.146|-0.0355|-0.163|0.00282|
||Obs.|2,862|1,274|1,588|1,814|1,048|
||Coeff.|-0.027|-0.065|0.006|-0.030|-0.028|
|Language - mention objects|(S.E.)|(0.053)|(0.067)|(0.065)|(0.062)|(0.063)|
|(SD)|Comp. mean|0.0333|-0.108|0.138|-0.0354|0.106|
||Obs.|2,862|1,274|1,588|1,814|1,048|
||Coeff.|0.068|0.059|0.075|0.055|0.082|
|Mth ti SD|(S.E.)|(0.054)|(0.077)|(0.061)|(0.067)|(0.068)|
|a - summaon ()|Comp. mean|-0.0407|-0.166|0.0521|-0.0730|-0.00628|
||Obs.|2,862|1,274|1,588|1,814|1,048|
||Coeff.|0.125*|0.108|0.143*|0.067|0.182*|
|Mth d b SD|(S.E.)|(0.058)|(0.083)|(0.067)|(0.068)|(0.073)|
|a - orer numers ()|Comp. mean|-0.0828|-0.214|0.0143|-0.0905|-0.0745|
||Obs.|2,862|1,274|1,588|1,814|1,048|
||Coeff.|-0.022|-0.012|-0.032|-0.020|-0.023|
|Abtt i SD|(S.E.)|(0.044)|(0.064)|(0.056)|(0.058)|(0.059)|
|srac reasonng ()|Comp. mean|0.0310|-0.102|0.130|-0.0317|0.0977|
||Obs.|2,862|1,274|1,588|1,814|1,048|

p<0.001***; p<0.01**; p<0.05* Note: Each cell-block is the result of a separate regression. Standard errors clustered at the village level are in parentheses. “Comp. mean” refers to the comparison group mean for the outcome variable. Column (1) regressions control for child characteristics (age, household size, household wealth asset index, parenting practices, mother's education and gender) and baseline child development measures (BMI, cognitive, fine motor, gross motor, and language). Columns (2) and (3) regressions use the same controls as column (1) except they exclude household wealth. Columns (4) and (5) regressions use the same controls as column (1) except they exclude parenting practices.

> 14 Appendix Table 6.

> 15 See Appendix Figure 2.

14
6. **Contrasting experiences – comparing project impacts for the younger and older cohorts** As described at the outset, the impact evaluation of the Indonesia ECED Project followed two cohorts of children. The focus of the paper so far has been on the younger cohort. The impacts of the intervention on the older cohort have previously been published (Brinkman et al. 2017b).

As shown in Figure 1, the data collection schedule meant that both cohorts of children were surveyed when they were age 5 and age 8. In this section, we contrast the impact estimates of these two cohorts side-by-side.<sup>16</sup> Table 5 presents the impact estimates at age 5 and age 8 for each cohort. Columns 1 and 4 are the results for the younger cohort previously shown in Tables 2 to 4 in this paper. Columns 2 and 5 present the equivalent results for the older cohort, previously reported in Brinkman et al., 2017b. Column 6 presents test score results for the older cohort, which have not previously been published.<sup>17</sup> Columns 3 and 7 show the results of a _t_ -test comparing the differences in impact between the two cohorts.<sup>18</sup> Overall, we find that the project impacts differed between the older and younger cohorts. The results are consistent with the fact that the _experiences_ of the older and younger cohorts were different along two key dimensions: (1) the phase of implementation when they were exposed to project playgroups and (2) the length of exposure to project playgroups at the appropriate age (3-4 years old). On the first dimension, children in the older cohort were exposed to the project in its first year whereas children in the younger cohort were exposed to a more mature project in its third year of implementation. On the second dimension, children in the older cohort were capped (at most) to one year of age-appropriate enrollment in project playgroups because they were already age 4 when the project began. In contrast, children in the younger cohort were only age 1 when the project began so they were more likely to enroll in project playgroups at the appropriate-ages of 3 to 4.

Table 5 shows that the younger and older cohorts had very different rates of enrollment in various types of early childhood education. Children from the younger cohort in treatment villages were 49.9 percentage points more likely to enroll in project playgroups compared to children from comparison villages. In contrast, the estimate for the older cohort was only 15.3 percentage points. This corresponds to a difference of 34.6 percentage points when the two cohorts were age 5. By age 8, the difference in impacts between the two cohorts had declined somewhat but was still very large. The effect on project playgroup enrollment for children in the younger cohort was 22.9 percentage points higher than that for the children in the older cohort.

For enrollment in non-project playgroups, the impact was a 22 percentage point decline for the younger cohort compared to 4.4 percentage point decline for the older cohort. By age 5, children in the younger cohort were 17.6 percentage points less likely to be enrolled in non-project playgroups compared to the older cohort. These differences in estimates were fairly similar by age 8, with a 15.2 percentage point difference between the two cohorts.

For kindergarten enrollment, the point estimates for the two cohorts were not

> 16 See Appendix B for the empirical strategy used to estimate the impacts for the 4 year-old cohort. 17 These are obtained using the approach described in Appendix B. The items used to construct scores for the 4 year-old cohort are identical to those used for the 1 year-old cohort.

> 18 Our test statistic to examine whether the estimates from the two cohorts are statistically different is � � � ����������������� �������� � ����������������� ������������������� ��� � ,���� � ���� � �������������������������� ����� � assuming 2Cov�β� , β��� � ��0��� . Since Cov�β� , β��� � ���� is typically positive, our assumption yields a conservative estimate of the test statistic.

15
distinguishable from each other – either by age 5 or by age 8. It is worth noting that by age 8, kindergarten enrollment had declined by 8.6 percentage points for the younger cohort and 13 percentage points for the older cohort. This suggests that the project may have encouraged children in treatment villages to substitute away from kindergarten relative to children in comparison villages, and this rate of substitution was not distinguishable across the two cohorts.

The younger cohort was also exposed to a higher dose of early childhood education than the older cohort. Relative to the older cohort, the younger cohort had accumulated 6 more months of project playgroups by age 5 and about 5 more months of project playgroups by age 8. This difference in enrollment duration between the two cohorts is likely driven by the fact that the older cohort was at the upper-end of the appropriate age for project playgroups at the beginning of the project. In contrast, the younger cohort was able to enroll in project playgroups at the appropriate age and remain in playgroups for longer.

The younger cohort enrolled in fewer months of non-project playgroups than the older cohort both by age 5 (3 months versus 0.6 months) and by age 8 (3.3 months compared to 0.9 months). Finally, we cannot reject the null hypothesis that the enrollment duration in kindergarten was significantly different across the two cohorts, both at age 5 and age 8.<sup>19</sup> Looking across the three types of services, the younger cohort was more likely to be enrolled in project playgroups and more likely to be enrolled for longer than the older cohort. As we turn to the results on child development and test scores, this fact should be borne in mind.

At age 5, the impacts on EDI outcomes varied across the two cohorts. Specifically, the younger cohort experienced an improvement in physical health and well-being (0.208 standard deviations) and emotional maturity (0.115 standard deviations). In comparison, the older cohort showed improvements in the domain of social competence (0.223 standard deviations). In particular, the treatment effect was significantly larger for the younger cohort than the older cohort in terms of physical health and well-being. However, for social competence, the treatment effect was significantly larger for the older cohort than the younger cohort.

At age 8, the magnitudes of the treatment effect on the EDI domains were smaller for the younger cohort compared to the older cohort in every dimension. However, the treatment effects for the younger cohort were not statistically different from the treatment effects for the older cohort. This was true across all 5 domains of the EDI.<sup>20</sup>

> 19 For the older cohort, the significant decline in kindergarten duration (-1.546) was consistent with the significant decline in kindergarten enrollment rate (-0.130). For the younger cohort, we did not find a significant decline in kindergarten duration but did find a significant decline in kindergarten enrollment rate (-0.086). This inconsistency for the younger cohort is due to the lack of precision in the estimates for months of enrollment. Overall, we interpret the results as evidence of substitution away from kindergarten in both cohorts.

> 20 In the case of communication skills and general knowledge domain the data suggest that treated children are in fact doing worse than comparison children (0.16 standard deviations). One explanation for this result are the ceiling effects shown in Appendix Figure 1.

16
# <u>Table 5. Comparison of impact estimates for the two cohorts</u>

||Tr|eatment effect at ag|e 5<br>Diff||Treatment|effect at age 8||
|---|---|---|---|---|---|---|---|
||Younger<br>cohort|Older cohort<br>(JOLE results)|erence.<br>between<br>cohorts|Younger<br>cohort|Older cohort<br>(JOLE results)|Older cohort<br>(new results)|<br>Difference.<br>between cohorts|
||(1)|(2)|(3)|(4)|(5)|(6)|(7)|
|Ever enrolled in project playgroup|0.499***|0.153***|0.346***|0.498***|0.269***||0.229***|
||(0.022)|(0.015)|(0.027)|(0.023)|(0.018)||(0.029)|
|Ever enrolled in non-project playgroup|<br>-0.220***|<br>-0.044**|<br>-0.176***|<br>-0.221***|<br>-0.069***||<br>-0.152***|
||(0.037)|(0.014)|(0.040)|(0.037)|(0.017)||(0.041)|
|Ever enrolled in kindergarten|<br>-0.075|<br>-0.058*|<br>-0.017|<br>-0.086*|<br>-0.130***||<br>0.044|
||(0.040)|(0.026)|(0.048)|(0.040)|(0.028)||(0.049)|
|Months enrolled in project playgroup|<br>7.781***|<br>1.914***|<br>5.867***|<br>8.704***|<br>3.855***||<br>4.849***|
||(0.396)|(0.158)|(0.426)|(0.414)|(0.254)||(0.486)|
|Months enrolled in non-project playgroup|<br>-3.001***|<br>-0.647***|<br>-2.354***|<br>-3.274***|<br>-0.943***||<br>-2.331***|
||(0.637)|(0.156)|(0.656)|(0.645)|(0.238)||(0.688)|
|Months enrolled in kindergarten|-0.199|0.590*|-0.789|-1.037|-1.546***||0.509|
||(0.436)|(0.270)|(0.513)|(0.664)|(0.444)||(0.799)|
|Physical health and well-being (SD)|<br>0.208***|<br>-0.026|<br>0.234*|<br>0.002|<br>0.104||<br>-0.102|
||(0.051)|(0.076)|(0.091)|(0.064)|(0.074)||(0.098)|
|Social competence (SD)|<br>0.018|<br>0.223**|<br>-0.205*|<br>-0.012|<br>0.024||<br>-0.036|
||(0.053)|(0.076)|(0.093)|(0.049)|(0.075)||(0.09)|
|Emotional maturity (SD)|<br>0.115*|<br>0.014|<br>0.101|<br>0.003|<br>0.158*||<br>-0.155|
||(0.056)|(0.071)|(0.090)|(0.045)|(0.068)||(0.082)|
|Language and cognitive development (SD)|0.073|0.128|-0.055|0.035|0.056||-0.021|
||(0.056)|(0.070)|(0.090)|(0.040)|(0.060)||(0.072)|
|Communication and general knowledge (SD)|<br>-0.005|<br>0.075|<br>-0.08|<br>-0.160***|<br>0.014||<br>-0.174|
||(0.074)|(0.079)|(0.108)|(0.061)|(0.132)||(0.145)|
|Language - match picture (SD)||||<br>0.133**||-0.100|<br>0.233***|
|||||(0.058)||(0.058)|(0.082)|
|Language - mention objects (SD)||||-0.027||-0.050|0.023|
|||||(0.053)||(0.054)|(0.076)|

17
# <u>Table 5. Comparison of impact estimates for the two cohorts</u>

||Tr|eatment effect at ag|e 5||Treatment|effect at age 8||
|---|---|---|---|---|---|---|---|
||Younger<br>cohort|Older cohort<br>(JOLE results)|Difference.<br>between<br>cohorts|Younger<br>cohort|Older cohort<br>(JOLE results)|Older cohort<br>(new results)|<br>Difference.<br>between cohorts|
||(1)|(2)|(3)|(4)|(5)|(6)|(7)|
|Math - summation (SD)||||0.068||-0.148**|0.216***|
|||||(0.054)||(0.051)|(0.074)|
|Math - order numbers (SD)||||0.125**||-0.155*|0.280***|
|||||(0.058)||(0.061)|(0.084)|
|Abstract reasoning (SD)||||-0.021||-0.124*|0.103|
|||||(0.044)||(0.057)|(0.072)|

p<0.001***; p<0.01**; p<0.05* Note: Each cell is the result of a separate regression. Standard errors clustered at the village level are in parentheses. Regressions for the younger cohort control for child characteristics (age, household size, household wealth asset index, parenting practices, mother's education and gender) and baseline child development measures (BMI, cognitive, fine motor, gross motor, and language). The difference between the treatment effects for the younger and older cohorts are estimated using a Welch's t-test or unequal variance t-test.

18
Lastly, we compared how the cohorts fared in terms of tests of language, mathematics and abstract reasoning in the early grades of primary school. The children in the younger cohort who live in treatment villages scored 0.133 standard deviations higher than children in comparison villages on a language test that asks them to match pictures with words. They also score 0.125 standard deviations higher than children from comparison villages on a math test that asks them to order numbers. In contrast, for the older cohort, children exposed to the treatment have similar scores in language tests when compared to children from comparison villages but lower scores on summation (-0.148), number ordering (-0.155) and abstract reasoning (-0.124).

Overall, children in the younger cohort performed better across a range of dimensions tested – the picture matching (0.233 standard deviations), ordering numbers (0.28 standard deviations), and summation (0.216 standard deviations). These results suggest that children with greater exposure to project playgroups, that is, children in the younger cohort, performed better in tests administered in primary schools than those in the older cohort who had significantly less exposure to project playgroups.

# **7. Cost-benefit analysis**

Before discussing these results in greater detail, we turn to another critical question: was the Indonesia ECED project a worthwhile investment? This section argues that it was. Comparable interventions in other countries range in cost from US$37 per child in India to US$289 in Colombia. The Indonesian project, on the other hand, cost approximately US$27 per child (all amounts in 2014 dollars).

Using the actual number of children reached by the project (673,162 as at June 2013) and the actual observed increase in educational attainment (0.1 years on average for the older cohort and 0.7 years for the younger cohort) allows us to present a rudimentary cost-benefit analysis (Table 6). It uses a conservative set of estimates of rates of return to education: which range from 6.8-10.6 percent as estimated by Duflo (2001) and from 6.1-12.3 percent as estimated by Patrinos et al. (2006). We assume that:

- a. there is a 6.5 percent rate of return to education (averaging the bottom end of the rates of return reported in the papers above in order to be more conservative in our analysis)

- b. children do not begin to realize the benefits of increased wages until age 18 c. they do so for 40 years

Under these assumptions, a 0.1-year increase in schooling results in a benefit-cost ratio of 0.65.<sup>21</sup> Similarly, a 0.7-year increase in schooling results in a benefit-cost ratio of 4.55. Using higher rates of return as assumed in the World Bank Project Appraisal Document (11.2 percent) suggests a correspondingly much higher benefit-cost ratio of 1.12. – 7.84. Thus even the most conservative cost-benefit estimates would suggest that the project did far better than breakeven. This is an underestimate of the benefit given the conservative estimates of returns to education used, the shorter-than-usual time horizon for accrual of benefits as well as the fact that these are only private returns for selected cohorts. Social returns to education have not been factored in, nor have any gains resulting from improved learning.

> 21 The 2012 GDP per capita in PPP terms was US$4,876. In our calculations of rate of return, we assume that rural wages are a third of this number.

19
Table 6: Cost-benefit-analysis

|Older cohort<br> <br> <br>|You<br>|nger cohort<br> <br>|
|---|---|---|
|Per beneficiary<br>($)<br>Total<br>($)<br>Per|benefic<br>($)|iary<br>Total<br>($)|
|Discounted stream of income (B)<br>96<br>64,573,510|601|404,598,373|
|Discounted cost (C)<br>76<br>51,469,388|76|51,469,388|
|B-C<br>19<br>13,104,122|525|353,128,985|
|Return for each USD invested<br>1.3||7.9|
|_Assumptions:_<br>Number of beneficiaries = 673,162.<br>Annual cost per beneficiary = USD 27.<br>Benefits start at age 18 and continue for 40 years.<br>Returns to education = 6.5%.<br>Discount rate = 5%.<br>Average annual earnings = 33% of 2012 GDP per capita in PPP terms.<br>Additional years of schooling for younger cohort = 0.7.<br>Additional years of schooling for older cohort=0.1.|||

# 8. **Discussion**

This paper showed that expanding access to preschools in rural Indonesia increased enrollment in preschools and improved child development outcomes. Most importantly, some of these positive impacts were sustained into primary school for the younger cohort. The evaluation design of the Indonesia ECED project allowed us to assess treatment effect variation across two cohorts of children who were exposed to key components of the project at different points in the project lifespan. Although the project had positive impacts for both cohorts of children, the fact that these impacts varied across cohorts suggests that the maturity and timing of preschool interventions are important factors to consider in impact evaluations.

There are a number of factors that explain the different outcomes resulting from expanded access to early childhood education services observed across the two cohorts in this evaluation.

The first is that children in the younger cohort were significantly more likely to enroll in project playgroups than the older cohort. The younger cohort was also enrolled in these services for longer than the older cohort. This was likely due both to the older cohort being at the upper end of the appropriate age range for playgroups at the onset of the project, as well as the maturity of the playgroups themselves at the time when the cohorts were able to enroll.

Another factor is how project playgroups evolved between 2009 and 2013 in terms of user fees. Approximately half of the project playgroups were not charging user fees at the beginning of the project, meaning that many of the children in the older cohort accessed the services for free (Brinkman et al. 2015).<sup>22</sup> By the end of the project, less than a quarter of the centers were free, with approximately half of all project playgroups charging between 10,000 and 25,000 IDR, which was comparable to the amount charged by non-project playgroups.

Among those children only enrolled in project playgroups, the wealth profile was very different for the two cohorts of children. On average, children in the younger cohort had

> 22 See Appendix Table 7.

20
higher wealth z-scores than those in the older cohort.<sup>23</sup> In conjunction with the introduction of fees as the project matured, the change in student composition in our data implies that it was easier for poorer children to enroll in project playgroups early in the project’s lifespan than in subsequent years.<sup>24</sup> Likewise, the quality of the preschools likely ebbed and flowed during the period under study. Brinkman et al. (2017a) establish the strong link between child development outcomes and the quality of the services being provided, as measured using classroom observation. Emerging evidence on how centers evolved during the life of the project and once project funding ended suggests that quality was not static (Hasan et al., forthcoming). Teacher training was delivered over time. Thus, at the outset of the project, centers began operating without necessarily having a full contingent of trained teachers. This process may have meant that centers had lower quality services during their first year than in later years of the project.

There are some limitations to this study. First, our analysis sample has attrition. Of the 3,089 children who were surveyed at baseline in 2009, 2,894 children (93.69%) were followed up for data collection in 2016. To limit attrition, enumerators were instructed to visit children in their homes if students and their primary caregivers were absent on the day of data collection in schools. As shown in Appendix Table 9, columns 1 and 2, there was no difference in attrition rates between treatment and comparison villages. Moreover, the interaction terms included in column 3 show that the characteristics of those who were not available for followup are largely similar across treatment and comparison villages. The table shows that children with less educated mothers and lower baseline gross motor skills were more likely to cease participation in treatment villages compared to similar children in comparison villages. However, overall, we do not find evidence that attrition poses a threat to the validity of our impact estimates.

Another limitation to our study is measurement error of our outcomes. As documented in Appendix Figures 1 and 2, several of our key outcome measures suffered from ceiling effects. This makes it difficult to detect effects that may have existed if an instrument that did not suffer from such effects had been used.

Lastly, we are unable to empirically test _why_ the treatment effect varies across the two cohorts. While we posed several plausible mechanisms, we cannot be sure why we found larger effects for the younger cohort than the older cohort at ages 5 and 8.

Despite these limitations, this study has meaningful findings. The results from this study indicate that a low-cost, community-based early childhood program can positively impact child development. For both cohorts, children who resided in treatment villages were more likely to enroll in project playgroups, were enrolled for longer, and had substantially better measures of child development than children in villages where these services were not available. The effects of this exposure persisted into early primary school for the younger cohort, as judged by tests of language, mathematics and reasoning.

There are a number of factors to consider when trying to ensure that the benefits of preschool programs are delivered consistently over time. This is particularly true in low-dose, center-based environments that are expanding in the developing world. As future early childhood education projects are designed and implemented, these myriad considerations will be important to balance against each other if sustained impacts are to be achieved.

> 23 See Appendix Table 8.

> 24 By the time the younger cohort was old enough to enroll in project playgroups, many more playgroups were charging fees. The introduction of fees was a direct response to the project funding coming to an end and the centers needing to devise an alternative sustainability strategy (Hasan et al., forthcoming).

21