metadata: authors: 0: author: Macdonald,Kevin Alan David
1: author: Brinkman,Sally Anne
2: author: Jarvie,Wendy
3: author: Machuca-Sierra,Myrna
4: author: Mcdonall,Kristen Andrew
5: author: Messaoud-Galusi,Souhila
6: author: Tapueluelu,Siosiana
7: author: Vu,Binh Thanh
count: Tonga
docna: 0: docna: Intervening at Home and Then at School : A
            Randomized Evaluation of Two Approaches to Improve Early
            Educational Outcomes in Tonga
docty: Policy Research Working Paper
subtopic: Educational Sciences,Effective Schools and Teachers,Educational Institutions & Facilities,Health Care Services Industry,Gender and Development
teratopic: Industry,Gender,Education
abstracts: cdata!: This paper evaluates and compares two
            randomized interventions in Tonga, one targeting the home
            environment of children up to age 5 and one targeting the
            school environment for first and second grade students. The
            first intervention supports communities to set up and run
            playgroups that aim to improve caregiver-child interaction
            at home and ultimately improve children's readiness for
            school. Among children of mothers without high school
            education, being in a treatment community positively affects
            the school readiness literacy domains and overall score by
            0.19 and 0.2 standard deviation, respectively, for girls and
            the literacy domains for boys by 0.17 standard deviation.
            The second intervention provides teachers with training,
            materials, and coaching to improve reading instruction
            practices and students’ reading ability. It increased
            average early grade reading scores by approximately 0.18
            standard deviation per year of exposure. Two cohorts of
            children were potentially exposed to both interventions,
            providing an opportunity to compare, for the same population
            of children, the effects of an early childhood intervention
            with a school-based intervention using a common measure of
            learning achievement. The school readiness intervention is
            found to have positive effects on early grade reading scores
            only among children in the reading intervention’s control
            group; effect sizes of approximately 0.29 standard deviation
            were found at the end of second grade for children exposed
            to one year of the school readiness intervention and at the
            end of first grade for girls exposed to two years of the intervention.
geo_reg_key: 517191
repnme: repnme: Intervening at Home and Then at School : A
            Randomized Evaluation of Two Approaches to Improve Early
            Educational Outcomes in Tonga
display_title: Intervening at Home and Then at
            School : A Randomized Evaluation of Two Approaches to
            Improve Early Educational Outcomes in Tonga
listing_relative_url: /research/2018/12/30709724/intervening-home-school-randomized-evaluation-two-approaches-improve-early-educational-outcomes-tonga
topicv3: Teachers Management m1326119 1523,Education, Skills Development and Labor
            Market m1326135 1484,Education m1327048 470,Gender m1327621 615,Health m1504661 3480
publishtoextweb_dt: 2018-12-19T00:00:00Z
historic_topic: Industry,Gender,Education
no_of_pages: 62
author_keys: 0: author_key: 962343
1: author_key: 925243
2: author_key: 166951
3: author_key: 959419
4: author_key: 1138320
5: author_key: 1261410
6: author_key: 1322496
7: author_key: 427112
model_extractions: WPS8682 Policy Research Working Paper 8682 Intervening at Home and Then at School A Randomized Evaluation of Two Approaches to Improve Early Educational Outcomes in Tonga _Kevin Macdonald Sally Brinkman Wendy Jarvie Myrna Machuca-Sierra Kris McDonall Souhila Messaoud-Galusi Siosiana Tapueluelu Binh Thanh Vu_ Education Global Practice December 2018
Policy Research Working Paper 8682

# **Abstract**

This paper evaluates and compares two randomized interventions in Tonga, one targeting the home environment of children up to age 5 and one targeting the school environment for first and second grade students. The first intervention supports communities to set up and run playgroups that aim to improve caregiver-child interaction at home and ultimately improve children’s readiness for school. Among children of mothers without high school education, being in a treatment community positively affects the school readiness literacy domains and overall score by 0.19 and 0.2 standard deviation, respectively, for girls and the literacy domains for boys by 0.17 standard deviation. The second intervention provides teachers with training, materials, and coaching to improve reading instruction practices and students’ reading ability. It increased average early grade reading scores by approximately 0.18 standard deviation per year of exposure. Two cohorts of children were potentially exposed to both interventions, providing an opportunity to compare, for the same population of children, the effects of an early childhood intervention with a school-based intervention using a common measure of learning achievement. The school readiness intervention is found to have positive effects on early grade reading scores only among children in the reading intervention’s control group; effect sizes of approximately 0.29 standard deviation were found at the end of second grade for children exposed to one year of the school readiness intervention and at the end of first grade for girls exposed to two years of the intervention.

This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/research. The authors may be contacted at tvu@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Intervening at Home and Then at School: A Randomized Evaluation of Two Approaches to Improve Early Educational Outcomes in Tonga**<sup>**1**</sup>

Kevin Macdonald, Sally Brinkman, Wendy Jarvie, Myrna Machuca-Sierra, Kris McDonall, Souhila Messaoud-Galusi, Siosiana Tapueluelu, Binh Thanh Vu Keywords: Human capital, randomized-controlled trial, literacy, primary education, early childhood development, cost effectiveness JEL Codes: I24, I25, I28, O15

> 1 This is an evaluation of two randomized pilot programmes implemented in Tonga and financed by the Global Partnership for Education through the World Bank’s Pacific Early Age Readiness and Learning Programme (PEARL).  We are grateful to Husein Abdul-Hamid, Felipe Barrera, Sachiko Kataoka, Toby Linden, Yohana Kristi, Harry Patrinos and Alanna Sincovich for their comments and contributions. The opinions expressed in this paper are those of the authors and not necessarily those of the World Bank Group.
# <u>1. Introduction</u>

Research identifying low-cost interventions to improve learning outcomes is especially pertinent to developing countries where human capital formation is often constrained by poor education quality and scarce resources to invest in education quality. One way to (loosely) categorize interventions to improve learning outcomes is into those that act through the school environment and those that act through the home environment. Research under the first category generally concentrates on pedagogic interventions including teacher training and learning materials as well as on school management (e.g., a recent review by Kremer et al. 2013), while the second category of research typically focuses on interventions to improve early stimulation or nutrition with parent education delivered through education or health centers, home visits or community groups (as in Berlinski & Schady 2015; Nadeau et al. 2011).

In this paper, we present an evaluation and comparison of two randomized interventions in the Kingdom of Tonga, one targeting the home environment and one targeting the school environment. The first intervention supports community leaders to establish volunteer-facilitated playgroups in which children up to 5 years old and their caregivers are exposed to guided play. The primary objective is to improve early stimulation through better caregiver-child interaction at home in order to improve school readiness. The second intervention provides 1<sup>st</sup> and 2<sup>nd</sup> grade teachers with training, lesson plans, materials and coaching on reading instruction. It aims to improve teaching techniques and early grade reading outcomes of students.

Both interventions were implemented as randomized-controlled trials, and measures of school readiness and early grade reading outcomes were collected. First, we find that the interventions had positive effects on the outcomes each intervention targeted. The school readiness intervention is found to have positive effects on school readiness for children with less educated mothers, especially girls. Positive effects on the prevalence of some types of caregiver-child activities at home are also found as well as effects on several indicators of community support for school readiness. The reading instruction intervention is found to have positive effects on early grade reading skills as well as on teachers’ teaching techniques. Second, we find that the school-readiness intervention, that targets the home environment, also affects subsequent reading 2
skills for select sub-populations. Exposure to both the school readiness intervention and the reading instruction intervention does not yield any additional effect, and the effects of the school readiness intervention are only found in the reading intervention’s control group. While the reading intervention has stronger effects on early grade reading outcomes, it is also more expensive per student; the school readiness intervention has relatively high cost effectiveness for the sub-populations it affects.

Our evaluation contributes to several strands of research. The effects of the school readiness intervention add to the emerging evidence on the effectiveness of playgroups. A closely related intervention in Indonesia targeting children aged 3 and 4 found positive effects on mathematics and language tests (Brinkman et al. 2015; Nakajima et al. 2016); though, it differs from the Tongan intervention by having formally trained local community members to be teachers rather than community volunteers to facilitate the groups. Additionally, the Tongan intervention actively focused on parents and their children participating in the groups, not just the children. Non-randomized evaluations of playgroups in Australia have also found positive effects on measures of school readiness (Gregory et al. 2016, Hancock et al. 2015). To our knowledge, our evaluation is the first randomized evaluation of play-based activities that are “community-led” in the sense that the activities are supported by community leaders and all participants including facilitators are community volunteers.

The school readiness intervention also contributes to research on how community participation can affect educational outcomes. In education research, community participation is typically studied from the perspective of school-based management and accountability (as in BarreraOsorio et al. 2009; Pradhan et al. 2014; Blimpo, Evans & Lahire 2015). However, in the theory behind playgroups’ effect on outcomes, community participation relates less to accountability and more to participatory or peer learning, analogous to how community prenatal and maternal women’s groups affect nutrition and health outcomes (e.g.: in Prost et al. 2013). This use of a community-based approach contributes also to the call for more research on different modalities of delivering early childhood development services (Dua et al. 2016; Shonkof et al. 2016).

Our findings on the impact of the reading intervention extend research examining effective 3
methods of in-service teacher training (for a recent review of evidence see Popova, Evans & Arancibia 2016). It builds on an emerging set of randomized-controlled trials showing the effectiveness of scripted lesson plans and teacher coaching on improving fundamental, early grade reading skills including phonemic awareness and oral reading fluency. These have been implemented in several challenging contexts in developing countries including Uganda, Liberia, Kenya (Piper & Korda 2011; Piper, Zuilkowski & Mugenda 2014; Lucas et al. 2014; Kerwin & Thorton 2015) and, as a remedial programme in Papua New Guinea (Macdonald & Vu 2018).

Finally, a distinct feature of our paper is the potential exposure of two birth cohorts to both interventions. While current evidence suggests that early interventions, including early stimulation, are more effective than later interventions (Nadeau et al. 2011), it is rare to be able to compare the effects of both an early childhood intervention and a school-based intervention on the same population of children. This is an important comparison because education policy makers, despite evidence on the effects of early childhood intervention on educational outcomes, tend only to consider school-based pedagogic interventions as a means to improve learning outcomes. From our experience, this reflects partly the legal mandate of education ministries but also an aversion to a perceived risk: education policy makers are often less familiar with the cost effectiveness of home interventions, especially those delivered outside of preschools, and view the effect on learning outcomes as being abstract or indirect. Our evaluation provides a tangible comparison of an early intervention with a school-based intervention by estimating the effects of both interventions on a common measure of learning achievement and a common population. Our evaluation demonstrates that an intervention targeting the home environment, delivered outside the school system, can be as effective on a per cost basis as a school-based pedagogy intervention but only for some sub-populations.

The paper is structured as follows. The next section presents information on the country context and descriptions of the interventions and research design. The effects of the interventions on their targeted outcomes are then discussed in succession followed by the effect of the school readiness intervention on reading outcomes.

4
# <u>2. Context, interventions and research design</u>

# _Country context_

Both interventions were implemented in the Kingdom of Tonga. The country is a small island state with a population of just over 100,000 inhabitants. While its population is small, the country consists of 176 islands of which 40 are inhabited and spans 718 square kilometers. The kingdom is divided into five main island groups: Tongatapu, Vava’u, ‘Eua, Ha’apai and the Niuas. The capital of Tonga, Nuku’alofa, is over 2,000 kilometers from its nearest large market, New Zealand, and over 3,000 kilometers from Australia.

The education system is quite small with approximately 17,000 students in primary school covering ages 6 to 11 and 14,500 secondary students aged 12 to 17. Enrollment rates in primary and secondary school are approximately 90 and 80 percent, respectively. Preschools are private, operated by either churches or communities, and data on preschool participation are limited, though UNESCO UIS reports approximately 2,000 students enrolled.

Despite this level of enrollment at the primary level, students’ learning outcomes remain a challenge. For example, assessments of early grade literacy in Tonga in 2009 found that only 30 percent of 3<sup>rd</sup> grade students were able to read fluently for comprehension (World Bank 2012a). Similar outcomes were found in other Pacific Island countries. In Vanuatu, approximately a quarter of 3<sup>rd</sup> grade students were able to read fluently for comprehension (World Bank 2012b, 2012c). In Kiribati and Tuvalu, 20 percent of 3rd grade children achieved minimum reading comprehension proficiency (World Bank 2017a, b). In Papua New Guinea, early grade reading assessments conducted in four provinces between 2011 and 2013 found that students lagged two years behind curriculum targets for fundamental pre-reading skills (World Bank 2014a, 2014b, 2014c, 2014d).

This poor performance in reading outcomes in Pacific Island countries is the primary motivation 5
for piloting the two interventions evaluated in this paper. The pedagogic intervention to improve reading instruction responds directly to the deficiencies found in the early grade reading assessments implemented in Tonga and other Pacific Island countries. The school readiness intervention emerged from a number of factors including substantial within-class variation in 1<sup>st</sup> grade reading performance, low levels of preschool participation, and few financial resources to expand preschool participation sustainably. The Tongan context is shared by other Pacific Island countries but also developing countries more generally, especially with remote populations.

# _School readiness intervention_

The school readiness intervention aims to affect young children’s early stimulation at home by improving child-caregiver interactions. The intervention is motivated by the importance of a young child’s physical, socio-emotional and cognitive development prior to school for his or her ability to learn in a school environment (Black et al. 2016; Nores & Barnett 2010). The intention of the intervention is to establish playgroups, termed community play-based activities (CPBAs), which consist of children up to age 5 and their caregivers. The CPBAs are led by volunteer facilitators who receive training and mentoring as part of the intervention. They occurred once or twice per week and lasted approximately two hours per session.

The intervention did not establish CPBAs directly; instead, it supported community leaders to initiate and maintain them. To help initiate CPBAs, the intervention team provided information and CPBA start-up materials to community leaders but also worked closely with them to engage community members to generate interest and support for the CPBA. This involved establishing a community education committee, finding facilitators, finding a venue for the activities which were normally town halls, churches or schools, and raising awareness of the CPBA among parents. The intervention team also worked closely with community leaders throughout the intervention period, helping to resolve problems including the loss of venue or facilitator. Ultimately the establishment of a CPBA in a treatment community relied on the community’s leader and members. In some treatment communities, CPBAs were not established or were established later during the intervention.

6
For this evaluation, the intervention is defined as this support provided by the intervention team which was randomized at the community level. The evaluation is at a community aggregate level rather than at the level of an individual child who attended a CPBA. It is not possible to identify the effect at the child level because exposure to CPBA is not random. Communities self-select by deciding whether to establish a CPBA, and parents self-select by choosing to send their children to a CPBA if it is established. For this reason, the evaluation measures the intentto-treat effect and underestimates the effect of CPBA exposure.

The intervention was expected to affect school readiness and subsequent learning outcomes in school through two stages. First, the intervention aimed to induce communities to establish and support a CPBA. Second, by participating in a CPBA, children were exposed to new learning opportunities and socialization, and caregivers were exposed to new types of play-based activities and interactions with their children that they could repeat at home, thus increasing stimulation for young children in the home environment.

The motivation for the intervention stems from several sources. First, previous research has established a link between early stimulation at home and child development, future educational outcomes and even labor market outcomes (e.g.: Gertler et al. 2014). Second, community group learning has been shown to be effective at affecting parenting behaviors, both in the context of playgroups (Brinkman et al. 2015; Nakajima et al. 2016) but also in nutrition and health as in community mothers’ groups for prenatal health and child nutrition (Prost et al. 2013; O’Rourke, Howard-Grabman, & Seoane 1998). The effectiveness of this approach to parent education reflects the benefits of participatory or peer learning and its ability to influence behaviors. Third, communities in Tonga are tight-knit and exhibit substantial social-capital (Farran 2009; Toganivalu 2008; Huffer 2006; Griffen 2006; World Bank 2013). This was expected to not only improve the effectiveness of peer learning approaches but also the ability of the communities to implement and sustain CPBAs.

The intervention began in mid-2015 and completed at the end of 2017; by June 2017, 1,337 children were participating. Because all children aged 0 to 5 in treatment communities are potential beneficiaries of the intervention, the recurrent cost per child per year was US$ 12.62.

7
This includes the cost of supporting community leaders, training and mentoring but excludes the time of volunteers and any support provided to the CPBA by community members.

# _Reading instruction intervention_

The reading instruction intervention, entitled “Come Let’s Read and Write” (CLRW), provided teachers in treatment schools with training and materials to implement a new pedagogic approach for reading instruction. The training and materials were coupled with monitoring and coaching, and offered to 1<sup>st</sup> and 2<sup>nd</sup> grade teachers. The approach aligns key learning competencies for basic reading and writing stipulated in the official Tongan curriculum with a greater degree of clarity on the sequence in which these skills should be taught.

Like the school readiness intervention, CLRW was expected to affect student reading outcomes in two stages. First, the intervention provided teachers with training, materials and coaching in order to induce them to adopt the new pedagogic approach. Second, students exposed to the new pedagogic approach were expected to have acquired better reading skills. For this paper, we refer to the former as the CLRW intervention and the latter as the CLRW pedagogic approach.

Evidence that the pedagogic approach can improve students’ reading ability stems from research on what skills children need in order to learn alphabetic languages (Wolf 2007; Linan-Thompson and Vaughn 2007; Sprenger and Charolles 2004; Chiappe & Wade-Woolley 2002; Gove & Cvelich 2011 and National Reading Panel 2000). These include, among others, an understanding of the relationship between printed letters and sounds (Scarborough 2002) and the speed at which a child can read (Fuchs et al. 2001; Abadzi 2006). CLRW is designed around the sequencing of skills and pedagogic methods based on this research (e.g.: August & Shanahan 2006; National Institute for Child Health and Human Development 2000; Pressley 1998; Snow, Burns & Griffin 1998).

In order to induce teachers to implement this structured pedagogic approach, the CLRW intervention relied on highly scripted lesson plans and periodic monitoring and coaching. Previous evaluations of similar interventions have shown positive effects on reading skills (Piper 8
& Korda 2011; Piper, Zuilkowski & Mugenda 2014; Lucas et al. 2014), and one study has shown a desired effect on teaching practices, measured by classroom observations (Kerwin & Thorton 2015). Both the scripted lesson plans and periodic coaching are seen as crucial to the effectiveness of the CLRW intervention. In-service training interventions in developing countries have generally yielded little evidence of impact, and one reason is a focus on knowledge rather than teaching techniques (Popova, Evans & Arancibia 2016).

The intervention was implemented in 2015 in first grade and extended to second grade in 2017 and 2018. 1<sup>st</sup> and 2<sup>nd</sup> grade teachers in 38 schools received training and materials, and the annual recurrent cost of the intervention was US$62.57 per student. This cost includes training, materials, monitoring and coaching, but it does not include the cost of developing the pedagogic approach, as this cost would not be incurred again if the intervention was scaled up.

# _Research design_

Both interventions were implemented as cluster randomized-controlled trials. For the school readiness intervention, 45 community clusters consisting of 59 _de facto_ communities and 66 statutory villages were randomly selected to receive the treatment while 45 other community clusters consisting of 83 statutory villages were randomly selected as control group communities. In a 2014 school readiness assessment, villages in Tonga were grouped into 129 communities and average school readiness scores were reported for each of these communities. For the intervention, 90 communities were randomly drawn from these 129. The communities were stratified by island group, and within island group, communities were implicitly stratified by the community’s number of children and the 2014 school readiness score. These 90 communities were then assigned to either the treatment or control group using systematic sampling, with communities ordered by their 2014 TEHCI score. This approach ensured that the 90 communities drawn to participate in the study had a similar number of children and school readiness score as those not selected, and that the treatment and control communities had similar school readiness scores at baseline.

For the CLRW intervention, 37 primary schools were randomly selected into the treatment group 9
and 36 primary schools were randomly selected into the control group. Selection was stratified by island group, school ownership (public or private), whether the school received students from communities included in the school readiness intervention, and the number of children in the first 6 grades. One school dropped out of the intervention after the second year, and this was replaced by a randomly selected school. Not all schools were eligible to be selected into the control or treatment groups: two other primary school interventions were being piloted in several schools which are excluded from the evaluation’s population. This included all schools on the island of ‘Eua. Note that schools in the treatment and control groups could include students from the school readiness intervention’s treatment communities, control communities and unassigned communities.

Data were collected on both child outcomes and behaviors that the interventions were trying to influence for both interventions. For the school readiness intervention, the Tongan Early Human Capabilities Index (TEHCI) survey measured school readiness and collected data on home activities, and surveys of treatment and control communities were conducted to measure community activities that supported school readiness. For the CLRW intervention, classroom observations were conducted to measure teaching practices and the Tongan Early Grade Reading Assessment (TEGRA) measured students’ early grade reading skill. These data are discussed in more detail in the sections on the respective interventions. Table 1 presents a timeline for the interventions and data collection by birth cohort. These cohorts are approximate, as children may enter first grade later and sometimes earlier than age 6.

As noted above for both interventions, there is a distinction between intervention and treatment. For the school readiness intervention, we are measuring the effect of providing community leaders with support to establish and maintain CPBAs, not the effect of a child’s exposure to a CPBA. For the reading intervention, we are measuring the effect of providing teachers with training, materials and coaching, not the effect of exposure to the new pedagogic approach. In both cases, the effect size of the intervention underestimates the effect size of the underlying treatment. As a result, the effects of the two interventions are not directly comparable primarily because all children in the CLRW treatment schools are exposed to the underlying treatment while a much smaller fraction of children in the community are exposed to a CPBA. However, 10
in the latter case, children are exposed to effects which arise outside the CPBA resulting from increased community support and awareness for school readiness.

|Table 1. Cohorts, intervention exposure, and surveys (excludes CPBA admin. and community data)|
|---|

|<br>**Year**|<br>**2014**|<br>**2015**|<br>**2016**|<br>**2017**|
|---|---|---|---|---|
|**Interventions**||School readiness<br>CLRW grades 1|School readiness<br>CLRW grades 1&2|School readiness<br>CLRW grades 1&2|
|**Surveys**|TEHCI Mar<br>TEGRA Nov||TEGRA Feb & Oct|TEHCI<br>TEGRA Mar & Oct|
|**Cohort born 2014**<br>Intervention<br>Data||Age 0<br>School readiness|Age 1<br>School readiness|Age 2<br>School readiness<br>TECHI|
|**Cohort born 2013**|Age 0|Age 1|Age 2|Age 3|
|Intervention<br>Data||School readiness|School readiness|School readiness<br>TEHCI|
|**Cohort born 2012**<br>Intervention<br>Data|Age 1|Age 2<br>School readiness|Age 3<br>School readiness|Age 4<br>School readiness<br>TEHCI|
|**Cohort born 2011**<br>Intervention<br>Data|Age 2|Age 3<br>School readiness|Age 4<br>School readiness|Age 5, Grade 1<br>CLRW<br>TEGRA|
|**Cohort born 2010**|Age 3|Age 4|Age 5, Grade 1|Age 6, Grade 2|
|Intervention<br>Data|TEHCI|School readiness|CLRW<br>TEGRA|CLRW<br>TEGRA|
|**Cohort born 2009**<br>Intervention<br>Data|Age 4<br>TEHCI|Age 5, Grade 1<br>CLRW|Age 6, Grade 2<br>CLRW<br>TEGRA|Age 7, Grade 3<br>TEGRA Oct.|
|**Cohort born 2008**<br>Intervention<br>Data|Age 5, Grade 1<br>TEHCI / TEGRA|Age 6, Grade 2|Age 7, Grade 3|Age 8, Grade 4|
|**Cohort born 2007**<br>Intervention<br>Data|Age 6, Grade 2<br>TEGRA|Age 7, Grade 3|Age 8, Grade 4|Age 9, Grade 5|

# <u>3. Effect of the school readiness intervention</u>

The school readiness intervention was expected to affect children’s school readiness primarily by improving caregiver-child interaction including through early stimulation in the household. The intervention supported community leaders to establish a CPBA, and if a CPBA was established, it provided training and mentoring to the volunteer facilitators. By participating in CPBAs, caregivers learn new types of interactions that they can do at home with their children that promote school readiness. Children participating in the CPBAs are also exposed to socialization and new learning opportunities. This section presents the effects of the school readiness 11
intervention. We find that the intervention positively affects school readiness outcomes for children with mothers who are less educated, especially girls. The intervention increases the prevalence of some home activities as well as measures of community support for school readiness.

# _Data_

Two sources of data are used to evaluate the school readiness intervention. First, community monitoring data were collected in treatment and control communities about their activities and support of early childhood care and education (ECCE). Information was collected about whether the community had meetings about ECCE, whether there was a community education committee, whether the committee had met, what type of support it received, whether there were playgroups or a preschool, and health services and activities, among others. Respondents included district officers, town officers, education committee members if one existed, and parents. However, because the purpose of the questionnaire was to monitor community support for playgroups in treatment communities and for ECCE more broadly in control communities, the questionnaires administered to treatment and control communities differ. Eight variables were identified as being relevant to the objectives of the intervention and compatible between the two questionnaires, these are the following: whether a preschool exists in the community, whether there is a play group including a CPBA, whether communities support ECCE services by encouraging parents, fundraising or by providing material resources, whether the community has an education committee, whether the community has a health center where the district nurse is located or centered, and whether the community has an education committee which includes the district nurse as a member.

The second set of data is the Tonga Early Human Capabilities Instrument (TEHCI) survey that measures school readiness for children aged 3 to 5 across a range of developmental domains. The survey was developed in 2013 by defining school readiness based both on the international evidence of the cognitive, socio-emotional and physical development needed to succeed in a school environment but also on Tongan values identified through consultations and workshops. The instrument measures development across eight developmental domains: physical 12
development, verbal skills, culture and spirituality, social and emotional development, perseverance, approaches to learning, numbers and concepts, and literacy skills. It is a rating type assessment where information about a child’s abilities and behaviors is obtained from individuals (including caregivers or teachers) that know the child well. The instrument is not designed to provide a diagnostic of a child’s development but rather to make inferences about developmental outcomes of a population.

The survey also collected data on the frequency of children being exposed to six different types of activities at home. These variables provide some insight into the home environment, though they do not provide information about the duration or quality of these caregiver-child interactions. The average was used as an index of home activities for the analysis.

The survey was conducted in 2014 prior to the interventions and in 2017 near the end of the interventions. In both rounds, all communities in Tonga were sampled and all children aged 3 to 4 who were identified by local informants including district nurses were sampled. All 5-yearolds were also sampled in 2014, but only those who were not in primary school were sampled in 2017. In this latter sample, the number of 5-year-olds is approximately 40 percent of the number of 3- or 4-year-olds. As a result, the 3- and 4-year-old sample is used to estimate the impact of the community school readiness interventions on school readiness outcomes and support for school readiness at children’s homes. Figure 1 presents a density plot of the average TEHCI score and the average score for the literacy sub-domains in standard deviations.

For estimates using both types of data, data are weighted by the inverse of communities’ selection probabilities, and standard errors are adjusted using a finite population correction to account for the small number of communities in Tonga from which the treatment and control communities were randomly drawn. Because the TEHCI survey samples all children in each community, there is no sampling variation within communities.

13
<!-- Start of picture text --> Figure 1. Density plots of average TEHCI score and literacy score (in<br>standard deviations)<br>average tehci score<br>: TMs ——— treatment<br>ana ——— control<br>-3 -2 -1 0 1 2 3<br>tehci literacy<br><!-- End of picture text --> <!-- Start of picture text -->
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5<br><!-- End of picture text -->
Table 2: TEHCI descriptive statistics and baseline balance (community level)

||all|treatment|control|difference|
|---|---|---|---|---|
|average TEHCI score|0.051|0.043|0.059|‐0.016|
||(0.024)|(0.038)|(0.031)|(0.049)|
|average literacy domains (SD)|0.063|0.046|0.08|‐0.035|
||(0.02)|(0.032)|(0.024)|(0.041)|
|home activities index|0.785|0.781|0.789|‐0.008|
||(0.007)|(0.01)|(0.009)|(0.013)|
|attending a preschool|0.415|0.401|0.43|‐0.029|
||(0.012)|(0.018)|(0.017)|(0.026)|
|female|0.464|0.479|0.449|0.03***|
||(0.005)|(0.007)|(0.008)|(0.01)|
|mother with high school edu.|0.596|0.579|0.615|‐0.036|
||(0.012)|(0.018)|(0.017)|(0.025)|

Standard errors included in parentheses. For differences, statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively.

# _Empirical strategy_

The randomized assignment of communities to receive the intervention allows us to identify the effect of the intervention; self-selection would require moving between communities which was unlikely. The effect of the intervention on school readiness was expected to arise through family participation in the CPBAs. We cannot estimate the effect of the CPBAs because communities self-select whether they establish a CPBA and families self-select participating in CPBAs. One approach would have been to use the randomized assignment of the intervention as an instrument for estimating the effect of the CPBAs; however, we believed the exclusion restriction would be violated. The establishment of the CPBA involved creating education committees and raising awareness in communities which could affect parenting for families that do not participate in CPBAs. As such, we have analyzed the data as intent to treat at the community level, In our model, the community intervention, T����, provided communities with support and 15
awareness to establish a community-based play activity, and it was expected to impact three sets of outcomes: (1) community factors including discussion of school readiness at community meetings and the establishment of CPBAs, (2) participation in CPBAs and caregiver-child activities at home, and (3) children’s school readiness. The effects of the school readiness intervention on each of these outcomes, Y��, are estimated by We are interested in how the school readiness intervention affects children from disadvantaged socio-economic groups. One indicator of this is mother’s education. The report of the 2014 TEHCI found that children of mothers without high-school education had lower school readiness measures (Brinkman & Vu: 31). More broadly, mother’s socioeconomic status has been linked with better educational outcomes (Bjorklund, Lindahl & Plug 2006), health and child development including obesity (Currie 2009), health for young adults (Lundborg, Nilsson & Rooth 2014), age of marriage and fertility (Breierova & Duflo 2004), among others. For a child’s participation in a CPBA, caregiver-child interaction, and school readiness, the effects of the school readiness intervention are estimated for different sub-populations defined by gender and whether the child had a mother with a high-school education or not. Children of mothers without a high-school education comprise 30 percent of our sample. The effects by subpopulation are estimated with the following model: where f is a binary variable for being a girl, and h is a binary variable for having a mother who completed high-school.

# _Effects of the school readiness intervention_

Table 3 presents the effect of the school readiness intervention on the measures of community support for school readiness collected in mid-2017. These are estimated with model (1). Data were collected previously, but this table is shown because it is the last round of data collected 16
and best reflects the intervention when fully implemented. Effects on the previous rounds of data collection are larger, as control communities had lower measures of support. Positive effects are found for whether the community had a playgroup or CPBA, whether the community supported ECCE through fundraising or donating materials, whether the community had an education committee, and whether the community had set up a health center for the district nurse. Treatment communities were slightly less likely to have an education committee that included the district nurse. Overall, the intervention had a positive effect on these measures of support for school readiness by the community. Note that these are not measures of implementation of the intervention, as the intervention does not guarantee change in these measures.

Table 3: Effect of the community intervention on community support for school readiness

||treatment|control|difference|
|---|---|---|---|
|Whether a preschool exists in the village|0.3|0.26|0.039|
||(0.034)|(0.032)|(0.047)|
|Whether there is a playgroup in the village|0.804|0.017|0.787***|
|(including CPBA)|(0.029)|(0.009)|(0.03)|
|Community supports ECCE services by|0.832|0.835|‐0.003|
|encouraging parents|(0.024)|(0.019)|(0.039)|
|Community supports ECCE services by fund|0.303|0.188|0.115**|
|raising|(0.033)|(0.024)|(0.047)|
|Community supports ECCE services by|0.217|0.135|0.082*|
|providing resources|(0.03)|(0.02)|(0.041)|
|Community has an education committee|0.896|0.559|0.337***|
||(0.022)|(0.033)|(0.041)|
|Community has a health centre where the|0.207|0.137|0.07*|
|district nurse is located|(0.03)|(0.025)|(0.039)|
|Community has an educ. cmtte. and the|0|0.017|‐0.017*|
|district nurse is a member|(0)|(0.009)|(0.009)|

Standard errors included in parentheses. For differences, statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively.

17
The effects of the school readiness intervention on the outcomes measured in TEHCI are presented in Table 4. The effects of the intervention on outcomes for all students are estimated using equation (1) and for the sub-populations of students using equation (2) by least squares. Full model estimates are presented in Annex Tables 1 and 2. Positive effects on overall school readiness were found only for girls of mothers with no high-school education; positive effects are found for both boys and girls of mothers with no high-school education on the average literacy domains. Annex Table 3 presents the effects for each TEHCI school readiness domain. For all children, positive effects were found only on the cultural and spiritual domain and the numeracy concepts domain. For girls of mothers without high-school education, the intervention also affected socio-emotional development, approaches to learning, and writing. Some negative effects were found for boys of mothers with high-school education.

Table 4. Effect of the school readiness intervention

||all|mothers wi<br>school ed|thout high<br>ucation|mothers<br>school e|with high<br>ducation|
|---|---|---|---|---|---|
|||girls|boys|girls|boys|
|average TEHCI score (SD)|0.04|0.2**|0.08|‐0.02|‐0.07|
||(0.06)|(0.1)|(0.08)|(0.07)|(0.05)|
|average TEHCI literacy domains (SD)|0.07|0.19**|0.17**|‐0.02|‐0.03|
||(0.05)|(0.08)|(0.07)|(0.06)|(0.05)|
|home activities index|0.03|0.01|0.05|0.01|0.02|
||(0.03)|(0.03)|(0.04)|(0.03)|(0.03)|
|attended a CPBA|0.21***|0.24***|0.2***|0.21***|0.19***|
||(0.02)|(0.03)|(0.03)|(0.02)|(0.02)|
|attended preschool|‐0.02|0.02|0.06**|‐0.07**|‐0.05**|
||(0.02)|(0.04)|(0.03)|(0.03)|(0.03)|

Standard errors included in parentheses. Statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively. Effects are estimated by equations (1) and (2), presented in Annex Tables 1 and 2. Effects are shown for children aged 3 and 4. Sample sizes are 3,429 children except for the home activities index model estimation which is 3,426 children.

No effects were found on the index of home activities; however, as shown in Annex Table 4, positive effects were found for individual home activities including reading to the child, singing songs, and naming or counting things. Positive effects were found for whether a child attended a 18
CPBA; this is nearly equivalent to the proportion of children attending a CPBA in treatment communities, as very few children in control communities attended a CPBA or playgroup. Approximately 21 percent of 3- and 4-year-olds in treatment communities attended CPBAs. Finally, the school readiness intervention had a positive effect on preschool participation for boys of mothers without high-school education but had a negative effect on those of mothers with high-school education.

# _Cost effectiveness_

The annual recurrent cost of the intervention was US$ 67,802.75. This included visits by the intervention team to support community leaders in setting up and running CPBAs as well as training and mentoring of volunteer facilitators. This cost excludes the opportunity cost of the facilitators’ time as well as any donations, either in-kind or monetary, provided by community members. Effect sizes of the school readiness intervention are per child, regardless of whether they attended a CPBA. All children in treatment communities aged 0 to 5 are potential beneficiaries. The number of children aged 0 to 5 in treatment communities was estimated to be 5,373 based on the 2017 TEHCI. The subsequent annual recurrent cost per child was US$ 12.62.

A common measure of cost effectiveness in the impact evaluation literature is standard deviations per US$100 (e.g.: Abdul Latif Poverty Action Lab 2014). Table 5 presents the effect sizes of the intervention in terms of a standard deviation per US$100 per child; this is equivalent to the effects presented in Table 4 divided by the cost per child in 100s of US$. 3- and 4-yearolds sampled in the 2017 TEHCI would have been exposed to the school readiness intervention for approximately two years. The cost per child of two years’ exposure was US$ 25.24. Positive effects were found on overall school readiness among girls of mothers with no high-school education and on the average of the literacy domains for both girls and boys of mothers with no high-school education. The effect size per 100 US$ for these sub-populations was 0.79, 0.75 and 0.67 standard deviations, respectively.

19
# _Discussion_

The school readiness intervention was found to have a positive effect on overall school readiness measured by the TEHCI for girls of mothers with no high-school education. It also affected the school readiness literacy domains of girls and boys in this group. Children of mothers without high-school education had lower levels of school readiness compared to children of higher educated mothers in the baseline TEHCI; hence, the school readiness intervention seems to have benefited the more vulnerable children.

Table 5. Effect size per 100 USD

||all|mothers w<br>school e|ithout high<br>ducation|mothers<br>school e|with high<br>ducation|
|---|---|---|---|---|---|
|||girls|boys|girls|boys|
|average TEHCI score|0.16|0.79**|0.32|‐0.08|‐0.28|
||(0.48)|(0.79)|(0.63)|(0.55)|(0.4)|
|average TEHCI literacy domains|0.28|0.75**|0.67**|‐0.08|‐0.12|
||(0.4)|(0.63)|(0.55)|(0.48)|(0.4)|

Standard errors included in parentheses. Statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively. Estimates are calculated by dividing the effects presented in Table 4 by the per child cost of the intervention in 100s.

No effects were found for the index of home activities, though positive effects were found for specific activities. Of course, the data do not provide a comprehensive picture of the quality of the activities or caregiver-child interaction more generally. Effects on school readiness likely reflect changes in the home environment that are not specifically captured in the TEHCI data. The lack of effect on school readiness for mothers with high-school education may be because the home environment is better for this sub-population, mitigating the effect of CPBA participation on the home environment. The six home activities are more prevalent among mothers with high-school education in the baseline, 2014 TEHCI.

While we cannot attribute these effects exclusively to CPBA participation, it is likely that CPBA is the main source of effect. Given the low participation in CPBAs, the effect of participating in a CPBA is likely quite high. For example, the intervention increased overall school readiness of 20
girls of mothers with no high-school education by 0.2 standard deviation; if this effect were exclusively the result of attending a CPBA, then the effect of CPBA attendance would be 0.83 standard deviation because 24 percent of these children attended a CPBA. Effect sizes may also be understated by the exclusion of 5-year-olds, as 5-year-olds at the time of the TEHCI survey would have had the most exposure to CPBA in 2017 and preceding years. Nevertheless, even with the low participation rate in CPBAs, just the intent-to-treat can have non-trivial effects on children with less educated mothers.

For boys of mothers with high-school education, the intervention reduced participation in preschool and had a small negative effect on some domains of school readiness. It is not clear why these two effects emerge. One hypothesis is that parents may have viewed CPBA as an alternative to preschool, and the reduction in preschool participation reduced school readiness. Preschools generally charge fees for participation while CPBAs did not. For boys of mothers without high-school education, the intervention increased preschool participation.

# <u>4. Effect of the CLRW intervention</u>

The CLRW intervention was expected to improve 1<sup>st</sup> and 2<sup>nd</sup> grade students’ reading skills by changing teaching techniques and exposing students to a new pedagogic approach. The randomized assignment of schools allowed us to identify the effect of the intervention on teaching practices measured in classroom observations and on reading outcomes. Self-selection could occur if students switched schools in response to the intervention; however, no evidence of this was reported. We found that the CLRW intervention had a positive effect on the prevalence of teaching practices promoted by the intervention and a positive effect on students’ reading abilities. Effect sizes tended to be higher for those exposed to two years of the intervention rather than just one.

# _Data_

Two sources of data were used to evaluate the effects of the CLRW intervention. The first was based on classroom observations that were conducted approximately halfway through the 2016 21
and 2017 school years in both CLRW treatment and control schools. Monitors attended reading instruction classes and assessed whether specific teaching practices and techniques promoted by the CLRW intervention were used by the teacher. Practices used by teachers during teaching and observed by the monitors were divided into 6 domains: phonemic awareness, phonics, reading of words, writing words and letters, forming of sentences and reading comprehension. For each of the practices within these domains, the observer rated whether the practice was fully, partially or not implemented. Classroom observations were conducted for all teachers in both 1st and 2nd grade of treatment and control schools. Estimates of effects described below account for unequal selection probabilities of schools and the small population of schools through a finite population correction.

The second source of data, the Tongan Early Grade Reading Assessment (TEGRA) was an adaptation of the Early Grade Reading Assessments initiated by the Research Triangle Institute (RTI International 2009) and adapted and applied to more than 100 languages in 65 countries (Dubeck & Gove 2015). The assessment was adapted to the Tongan language prior to its first implementation in 2009 and updated for subsequent rounds in 2014, 2016 and 2017. The assessment measured reading skills across 9 domains. Data were also collected on several measures of home support for reading.

The 2016 and 2017 rounds of TEGRA were used in this analysis to measure the impact of CLRW on students’ acquisition of reading skills during the school year. The survey was conducted in February and October of 2016, corresponding to the beginning and end of the 2016 school year, and in March and October 2017, corresponding to the beginning and end of the 2017 school year. All treatment and control schools were sampled. Within each school, all first and second grade classes were selected within which a random sample of students was drawn, stratified by gender and, for first grade, whether they had attended a CPBA previously. For the October 2017 round, third grade classes were also selected. For this analysis, standard errors were estimated to be robust for intra-cluster correlation within schools (the primary sampling unit) and adjusted for small population size both at the school stratum and student stratum levels. Sampling weights were calculated to account for unequal selection probabilities of schools and attrition of schools and students.

22
TEGRA provided data on nine reading domains; a list of these is included in the annex. An average reading score is used to measure the overall effect of the CLRW intervention. It was defined as the average of the standardized scores of the nine reading domains. The reading domains were standardized using the mean and standard deviation of the baseline sample for each particular grade and round of survey, as some reading domains were not comparable between rounds. An index of home support was also used in this analysis and defined as the average for the 7 measures of home support. These measures of home support were whether a student receives help with his or her homework, whether parents are interested in his or her school day, whether he or she reads aloud at home, whether someone reads to the student at home, whether the child has been absent in the past two weeks of school, and whether he or she attended preschool and a CPBA. Note that the student is the respondent to these questions and is typically 6- or 7-year-old; it is not clear how reliable their responses were.

Three samples were used to estimate the effects of the CLRW intervention. The first sample was used to measure the average effect of one year of exposure to the intervention. This consisted of 1<sup>st</sup> grade students in 2016 and 2017. Both cohorts were weighted equally to provide an average effect. 1<sup>st</sup> grade students in 2015 were excluded from this because no data were collected at the beginning or end of the 2015 school year. The second sample was used to measure the effect of two years of exposure. This consisted of students who started 1<sup>st</sup> grade in 2016. The third sample was used to measure the persistence of the CLRW intervention’s effect on reading skills. This consisted of students who were in 1<sup>st</sup> grade in 2016 but compared their test scores at the end of the 2016 school year (at the end of two years of exposure to the intervention) with those at the end of the 2017 school year, providing an estimate of the effect of being out of the intervention for one year.

# _Descriptive statistics and baseline balance_

Table 6 presents baseline means and balance of three variables: average TEGRA score, whether a child was female, and the index of home support. Figure 2 plots the density of average TEGRA score and Annex Figure 1 plots the densities for all 9 reading domains. The sample 23
used to measure the persistence of effect did not have baseline data, as no data were collected for this cohort prior to the intervention. For both interventions, small, statistically significant differences between treatment and control schools were found for average TEGRA achievement and gender; treatment schools tended to have slightly lower baseline achievement and a slightly lower proportion of female students.

Table 6. TEGRA descriptive statistics and baseline balance by sub‐sample

||all|treatment|control|difference|
|---|---|---|---|---|
|_1 year exposure sub‐sample (stu_|_dents starting_|_1st grade in 20_|_16 and 2017_|_)_|
|average TEGRA score (SD)|‐0.036|‐0.064|0|‐0.064**|
||(0.013)|(0.018)|(0.017)|(0.025)|
|female|0.44|0.413|0.475|‐0.062***|
||(0.004)|(0.003)|(0.006)|(0.007)|
|home support index|0.528|0.521|0.538|‐0.017|
||(0.006)|(0.008)|(0.007)|(0.011)|
|_2 year exposure sub‐sample (stu_|_dents starting_|_1st grade in 20_|_16)_||
|average TEGRA score (SD)|‐0.044|‐0.072|0|‐0.072**|
||(0.014)|(0.018)|(0.024)|(0.03)|
|female|0.407|0.367|0.471|‐0.104***|
||(0.005)|(0.003)|(0.008)|(0.008)|
|home support index|0.528|0.521|0.538|‐0.017|
||(0.006)|(0.008)|(0.007)|(0.011)|

Standard errors included in parentheses. For differences, statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively.

# _Attrition_

Table 7 presents attrition rates between treatment and control schools for samples used to estimate 1 and 2 years of exposure to the CLRW intervention. For the 1-year exposure sample, the attrition is 13 percent overall. Treatment schools have 4.4 percent lower attrition. Attrition rates are higher for the 2-year exposure sample at 33 percent; attrition in treatment schools is 8 percent lower. Comparison of baseline average reading achievement between attritors and nonattritors is presented in Annex Table 5; attritors tend to have lower reading achievement than 24
Figure 2. Density plots of average TEGRA score by sample and treatment status (in standard deviations) <!-- Start of picture text --> baseline<br>/ ------- sample 1 control<br>/ a.x ------- sample——— sample 21 treatmentcontrol<br>“ Va ; NY — sample 2 treatment<br>-0.5 0 0.5 1<br>endline<br>sr id ~ Sa<br>-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5<br><!-- End of picture text --> Sample 1: 1 year exposure sub-sample Sample 2: 2 year exposure sub-sample CLRW P
<!-- Start of picture text --> p=Bo+B? crew+ U3<br>R=BOt+BL eR +82 crrwtUs<br>10<br>RR<br>R=BO+Br R +B corwt+B3ft+Be crrwtUs<br>R-BL R Bi<br>4<br>1<br><!-- End of picture text -->
Table 7. TEGRA attrition rates and balance of the non‐attrition sample by sub‐sample

||all|treatment|control|difference|
|---|---|---|---|---|
|Attrition rates<br>|||||
|_1 year exposure sub‐sample_|||||
|sample size at baseline|2540|1268|1272|n.a.|
|sample size at end‐line|2199|1226|1073|n.a.|
|attrition rate (pop. est)|0.13<br>(0.006)|0.111<br>(0.008)|0.155<br>(0.008)|‐0.044***<br>(0.012)|
|_2 year exposure sub‐sample_|||||
|sample size at baseline|1239|620|619|n.a.|
|sample size at end‐line|818|383|184|n.a.|
|attrition rate (pop. est)|0.331|0.299|0.381|‐0.081***|
||(0.013)|(0.017)|(0.018)|(0.025)|
|Non‐attrition sample baseline balance|||||
|_1 year exposure sub‐sample_|||||
|average TEGRA score (SD)|‐0.026|‐0.059|0.018|‐0.078***|
||(0.014)|(0.019)|(0.019)|(0.027)|
|female|0.44|0.414|0.475|‐0.061***|
||(0.004)|(0.004)|(0.007)|(0.008)|
|home support index|0.526|0.52|0.538|‐0.018|
||(0.006)|(0.009)|(0.008)|(0.012)|
|_2 year exposure sub‐sample_|||||
|average TEGRA score (SD)|‐0.034|‐0.064|0.021|‐0.085**|
||(0.017)|(0.018)|(0.032)|(0.038)|
|female|0.423|0.392|0.478|‐0.086***|
||(0.006)|(0.008)|(0.01)|(0.012)|
|home support index|0.537|0.536|0.539|‐0.003|
||(0.007)|(0.009)|(0.009)|(0.013)|

Standard errors included in parentheses. For differences, statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively.

# _Effect of the CLRW intervention_

The effect of the CLRW intervention on the percent of teachers fully implementing the teaching practices promoted by the intervention is presented in Table 8. The effects for each category of teaching practice are presented in Annex Table 6. Statistically significant, positive effects were 27
found both for all teaching practices as well as all sub-domains of teaching. In most domains, effect sizes tended to be higher in the 2016 classroom observations than the 2017 classroom observations. The proportion of teachers in control schools fully implementing the teaching techniques increased between 2016 and 2017.

Table 8. Effect of CLRW on the proportion of teachers fully implementing all teaching practices targeted by the intervention

||Treatment|Control|Difference|
|---|---|---|---|
|School year 2016|0.926|0.518|0.407***|
||(0.006)|(0.019)|(0.02)|
|School year 2017|0.89|0.623|0.267***|
||(0.007)|(0.017)|(0.019)|

Standard errors included in parentheses. For differences, statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively.

Table 9. Effect of CLRW on average TEGRA reading scores (standard deviations)

||Effect on s<br> <br>|tudents' aver<br>reading score<br>|age TEGRA<br>s<br>|
|---|---|---|---|
||1 year<br>exposure|2 years'<br>exposure|1 year after<br>2 years of<br>exposure|
|all students|0.19***|0.33***|‐0.09**|
||(0.04)|(0.06)|(0.04)|
|girls|0.18***|0.34***|‐0.11***|
||(0.05)|(0.06)|(0.04)|
|boys|0.21***|0.38***|‐0.04|
||(0.04)|(0.08)|(0.05)|
|difference in girls' and boys' effects|‐0.03|‐0.04|‐0.07|
||(0.04)|(0.08)|(0.05)|
|sample size|2199|818|862|

Standard errors included in parentheses. Statistical significance at the 1, 5 and 10 percent levels denoted are by ***, **, and *, respectively. Effects are estimated by equations (4) and (5), presented in Annex Table 7.

28
Table 9 presents the effects of the CLRW intervention on average TEGRA reading score; these effects were based on the estimates of equations (4) and (5) presented in Annex Table 7. Annex Table 8 presents effects for each domain of TEGRA. Positive effects of 1- and 2-years’ exposure to the CLRW intervention were found for all students, and no difference in effects was found between boys and girls. Effect sizes are larger for 2 years of exposure. For the 1-year exposure sample, effects were largest for the initial sounds, letter sounds, reading and listening comprehension domains. Positive effects were not found for the remaining domains. Two years of exposure to the intervention yielded positive effects in all domains except letter names and listening comprehension.

Table 10. Lee bound estimates of impact of CLRW on average TEGRA score

|||1 year exposure<br>sample|2 year exposure<br>sample|
|---|---|---|---|
|All students||||
||lower bound|0.09**|0.17***|
|||(0.04)|(0.06)|
||upper bound|0.24***|0.51***|
|||(0.04)|(0.06)|
|Girls||||
||lower bound|0.07|0.15***|
|||(0.05)|(0.05)|
||upper bound|0.24***|0.55***|
|||(0.05)|(0.07)|
|Boys||||
||lower bound|0.11***|0.2**|
|||(0.04)|(0.07)|
||upper bound|0.25***|0.48***|
|||(0.03)|(0.07)|

Lee (2009) tight bounds are presented. The grouping variables for the 1 year exposure sample are cohort and gender, and for the 2 year exposure sample the grouping variable is gender. Gender is not used as a grouping variable for the bounds estimated for boys and girls. Standard errors included in parentheses. Statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively. Standard errors are estimated using jackknife replicates.

29
One year after 2-years of exposure, a negative effect was found for girls but not boys, suggesting that the advantage CLRW provides students may decline for girls; however, the negative effect is smaller than the effect size for either 1 or 2 years of exposure.

Lee (2009) bounds are presented in Table 10 for the effect on average TEGRA achievement. These are “tight” bounds and use gender, and for the 1-year exposure sample, cohort as grouping variables. The student’s cohort is also used to tighten the bounds estimated for gender for the 1- year exposure sample. In both cases, the lower bound estimates were positive and statistically significant except for the effect of 1-year of exposure on girls.

# _Cost effectiveness_

The intervention’s costs consisted of teacher training workshops, materials and the periodic coaching and mentoring. The annual costs of these totaled US$87,469.05 for the 37 schools in the intervention. At baseline, 1,398 1<sup>st</sup> and 2<sup>nd</sup> grade students attended the treatment schools. The cost per student was US$ 62.57. Table 11 presents effect sizes of 1 and 2-years of exposure per US$100. For all cases, the effect sizes per US$100 are approximately 0.3 standard deviation. While two years of exposure yields higher effect sizes, it is also twice as expensive; hence, the cost effectiveness is slightly less than that of 1 year of exposure.

Table 11. Effect of CLRW TEGRA reading skills per 100 USD

||1year exposure sample|2years exposure sample|
|---|---|---|
|all|0.3***|0.26***|
||(0.06)|(0.05)|
|girls|0.29***|0.27***|
||(0.08)|(0.05)|
|boys|0.34***|0.3***|
||(0.06)|(0.06)|

Standard errors included in parentheses. Statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively. Estimates are calculated by dividing the effects presented in Table 9 by the per student cost of the intervention in 100s.

30
# _Discussion_

The evaluation finds positive effects both on teachers’ teaching practices and average TEGRA achievement. This is consistent with the intervention’s expected effect on reading outcomes by first improving the school environment. It is not clear why the percent of teachers fully implementing several of the teaching techniques promoted by the intervention increased in control schools between 2016 and 2017. It may have been a form of contamination in which treatment school teachers were sharing the techniques they had learned with control school teachers; however, control school teachers did not receive the periodic monitoring and coaching that treatment school teachers received, and with the exception of one known case, did not receive the materials. Additional analysis of the data did not find any statistically significant difference in the effect of CLRW on reading outcomes between 2016 and 2017. If there had been some contamination of the control group in 2017, it did not seem to affect the outcomes.

In terms of internal validity, some minor imbalance between treatment and control schools was found in the baseline average TEGRA achievement and gender composition. Our decomposition of effect sizes into gender groups and controlling for baseline average achievement accounts for this imbalance. The validity of our study is also complicated by the high rate of attrition in our samples, especially for the 2-year exposure sample. However, the non-attrition samples did not exhibit a substantially different imbalance than the baseline sample, and the lower Lee bound estimates remained positive and statistically significant except in one case.

The intervention’s cost effectiveness was found to be approximately 0.3 standard deviation per US$100. The cost effectiveness of 2 years of exposure was approximately the same as 1 year of exposure.

Finally, the treatment and control school sample’s population excluded several schools in Tonga because the government was piloting another intervention. However, it is unlikely that these excluded schools were systematically different. We do not believe that the exclusion of these schools would have had a substantial effect on the external validity of the study.

31
# <u>5. Comparing the interventions’ effects on reading skills</u>

The school readiness intervention ultimately aimed to improve the home environment and subsequently improve a child’s ability to learn in a school environment. While its objectives were much broader than improving the specific reading skills assessed in TEGRA, an important question is did it have any effect on them? Two birth cohorts were exposed to both the school readiness intervention prior to starting school and the CLRW intervention once in school. Using data on mappings between communities and schools, we estimate the effect of the school readiness intervention on early grade reading scores. We find a positive effect for children with one year of exposure to the school readiness intervention at the end of two years of schooling in CLRW control schools and for girls with two years of exposure to the school readiness intervention at the end of one year of schooling.

# _Data_

In TEGRA, there are two sources of data on whether a student lived in a treatment or control community prior to starting primary school. First, the February round of the 2016 TEGRA collected data on which community 1<sup>st</sup> grade children lived prior to starting school. However, the information could not be collected for 17 treatment or control schools, and a further review of the data found that it may not have been reliable for all students. Second, 421 students in TEGRA were previously sampled in the 2014 TEHCI data. This source of data of where a child lived prior to primary school was reliable and almost all schools in TEGRA contained some of these students. We use this source to derive for each school the probability that a student was from a school readiness treatment community as opposed to a control community or unassigned community. Unassigned communities were also valid controls as the treatment (and control) communities were selected at random from all communities in Tonga. The indicator was calculated as the proportion of students in a school who came from a CPBA treatment community. This measure was highly correlated with an analogous indicator calculated using the first source of data on students’ communities.

32
A second variable we added was an estimate of the average baseline 2014 school readiness score for students in each school. This was calculated as the average baseline school readiness score for the students in the school that were linked to the 2014 TECHI. This variable was used to establish balance in baseline TEHCI school readiness between treatment and non-treatment students using our mapping approach. An analogous variable was defined using the TEHCI home activity index.

We defined three sub-samples of interest for estimating the effect of the school readiness intervention on TEGRA achievement. The first sub-sample consisted of 1<sup>st</sup> grade students sampled at the beginning and end of the 2016 school year, and the second sub-sample consisted of 1<sup>st</sup> grade students sampled at the beginning of the 2016 school year and end of the 2017 school year. Students starting first grade in 2016 would have been exposed to the school readiness intervention in the year prior to starting primary school. This cohort would have been mostly born in 2010. These two sub-samples are used to estimate the effect of one year of exposure to the school readiness intervention on 1 and 2 years of schooling, respectively. The third subsample consisted of students in 1<sup>st</sup> grade during the 2017 school year. They would have been exposed to the school readiness intervention in the two years prior to starting primary school and been born mostly in 2011. This sub-sample was used to estimate the effect of two years of the school readiness intervention and one year of schooling. It should be noted that 2015 was the implementation year of the school readiness intervention, and playgroups did not become established until approximately halfway through the year.

# _Empirical approach_

Let _i_ and _j_ index the _i_<sup>th</sup> student in the _j_<sup>th</sup> school, and let T��,�� denote whether the student came from a school readiness treatment community or not. We defined the following model for each school, _j_ , as where u�,�� is a mean-zero, student-level error term and v�,� is a school-level error term. If P��,� 33
<!-- Start of picture text --> Rj = BO + Bi YR) +BZ srg + Bs corwyj + %,;<br>R YR<br>76<br>33<br>SR,j<br>6,j<br>R=BOt+BE R+BS set+B3 crew tus<br>R=BO+B? R+B3 se t+B3 cornwt+Bif+Be sret+Be crew<br>+B? sr crew t+Bs sr crrw+ Uo<br>“sp = Boo + Bt° sr +10<br><!-- End of picture text -->
where Y<sup>�</sup> ��� denotes estimated average baseline school readiness for the communities mapped to the students in the school. A non-zero estimate of β��� implies that imbalance exists between the school readiness treatment and control group students using this method of mapping. Equation (10) is also estimated using gender, the TEHCI home activity index, exposure to the CLRW intervention, and attrition as dependent variables, to measure imbalance in these variables as well.

Table 12. Estimates of equation (10) testing balance for the baseline sample

|dependent variable:|baseline<br>community<br>TEHCI<br>score (SD)|home<br>support|female|CLRW|
|---|---|---|---|---|
|_1 year of school readiness interv_<br>|_ention sample_<br>|_(starting 1st_<br>|_grade in 201_<br>|_6)_<br>|
|probability of being from a|‐0.06|‐0.02|0|0.08|
|treatment community|(0.08)|(0.01)|(0.01)|(0.09)|
|constant|‐0.01|0.78***|0.41***|0.58***|
||(0.04)|(0.01)|(0.01)|(0.05)|
|observations|1214|1214|1214|1214|
|r‐square|0.00|0.01|0.00|0.00|
|_2 years of school readiness interv_|_ention sample_|_(starting 1st_|_grade in 20_|_17)_|
|probability of being from a|‐0.09|‐0.02|0.01|0.11|
|treatment community|(0.07)|(0.01)|(0.02)|(0.09)|
|constant|0|0.78***|0.47***|0.47***|
||(0.03)|(0.01)|(0.01)|(0.05)|
|observations|1290|1290|1289|1290|
|<sup>r‐square</sup>|0.01|0.01|0.00|0.01|

Table presents estimates of equation (10) for the listed dependent variables at baseline. Non‐zero coefficients of the probability of being from a treatment community imply imbalance between students from treatment and non‐treatment communities as a result of the community‐school mapping. Standard errors included in parentheses. Statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively.

35
# _Baseline balance_

Table 12 presents estimates of equation (10) for the 1-year exposure sample and the 2-year of exposure samples as well as four different dependent variables, in order to test for imbalance between students exposed to the school readiness intervention and those not. Note that the samples used to estimate the effect of one year of exposure to the school readiness intervention on one and two years of schooling are the same at baseline; they started in 1<sup>st</sup> grade in 2016. No statistically significant association was found between the proportion from a school readiness intervention community and any of the four baseline variables. This suggests no evidence of imbalance.

# _Attrition_

Equation (10) is also estimated using attrition as a dependent variable to test whether there were differences in attrition rates between school readiness treatment and control groups. We find that the school readiness intervention had a negative effect on attrition for only one sub-sample: that used to estimate the effect of one year of exposure to the school readiness intervention on reading scores after one year of schooling. Baseline balance for the non-attrition sample is tested analogously using equation (10) again, and no imbalances were found for any of the subsamples.

# _Effect of the school readiness intervention on reading achievement_

Table 14 presents the estimated effects of the school readiness intervention disaggregated by gender and CLRW treatment exposure. Model estimates are presented in Annex Table 8. The school readiness intervention had a positive effect on girls’ reading achievement after 1 year of exposure to the school readiness intervention and 2 years of schooling as well as after 2 years of exposure to the school readiness intervention and 1 year of schooling, in CLRW control schools only. For this latter group, the effect of CLRW on the effect of the school readiness intervention was negative; being exposed to the CLRW intervention reduced gains from exposure to the school readiness intervention. A positive effect for boys in the CLRW control school after 1 36
year of exposure to the school readiness intervention and 2 years of schooling was also found.

# _Cost effectiveness_

The annual cost per child of the school readiness intervention was US$12.62 per child and for the CLRW intervention, US$62.57 per student. Table 15 provides a comparison of the cost effectiveness of the two interventions. These are based on the effect sizes of exposure to one intervention conditional on no exposure to the other. For example, the effect per US$100 of the school readiness intervention was based on the effect of the school readiness intervention for students in the CLRW control group presented in Table 14, and the effect per US$100 of the CLRW intervention was conditional on there being a zero probability of being from a school readiness intervention community. These figures are based on linear combinations of the estimated coefficients of equation (9) presented in Annex Table 9.

Because the per child cost of the school readiness intervention is much smaller than the per child cost of the CLRW intervention, the effect sizes per US$100 tend to be higher for the school readiness intervention compared to the CLRW intervention. However, the school readiness intervention has positive effects that are statistically significant for three sub-samples, as was presented in Table 14; the CLRW intervention affects reading for all the sub-populations presented here. For one sub-population, girls exposed to one year of the community school readiness intervention and two years of schooling in a CLRW control group, the costeffectiveness of the school readiness intervention was higher than that of CLRW. For all other sub-populations, no statistically significant difference in cost effectiveness was detected.

37
Table 13. Estimates of equation (10) testing balance in attrition rates and testing balance for the non‐attrition sample

||||non‐attritio|n sample||
|---|---|---|---|---|---|
|dependent variable:<br>_1 year of school readiness interve_|attrition<br>_ntion 1st gr_|baseline<br>community<br>TEHCI<br>score (SD)<br>_ade sample_|home<br>activity<br>index|female|CLRW|
|probability of being from a|‐0.05**|‐0.06|‐0.02|0|0.08|
|treatment community|(0.02)|(0.08)|(0.01)|(0.01)|(0.09)|
|constant|0.14***|‐0.01|0.78***|0.41***|0.58***|
||(0.01)|(0.04)|(0.01)|(0.01)|(0.05)|
|observations|1214|1049|1049|1049|1049|
|r‐square|0.00|0.00|0.01|0.00|0.00|
|<br>_1 year of school readiness interve_|_ntion 1st &_|_2nd grade sam_|_ple_|||
|probability of being from a|‐0.02|‐0.04|‐0.02|‐0.01|0.08|
|treatment community|(0.03)|(0.07)|(0.01)|(0.02)|(0.09)|
|constant|0.25***|‐0.01|0.78***|0.41***|0.58***|
||(0.02)|(0.04)|(0.01)|(0.01)|(0.05)|
|observations|1214|925|925|925|925|
|<sup>r‐square</sup>|000|000|001|000|000|
|<br>_2 years of school readiness interv_|.<br>_ention 1st g_|.<br>_rade sample_|.|.|.|
|probability of being from a|‐0.02|‐0.09|‐0.02|0.01|0.11|
|treatment community|(0.02)|(0.07)|(0.01)|(0.02)|(0.09)|
|constant|0.12***|0|0.78***|0.47***|0.47***|
||(0.01)|(0.03)|(0.01)|(0.01)|(0.05)|
|observations|1290|1140|1140|1140|1140|
|<sup>r‐square</sup>|0.00|0.01|0.01|0.00|0.01|

Table presents estimates of equation (10) for attrition at baseline and for the listed dependent variables at baseline for the non‐attrition sample. Non‐zero coefficients of the probability of being from a treatment community imply differential attrition rates or, for the non‐attrition sample, imbalance between students from treatment and non‐treatment communities as a result of the community‐school mapping. Standard errors included in parentheses. Statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively.

38
Table 14. Effect of the school readiness intervention and CLRW treatment by <u>gender</u>

||1 year of sch<br>interventi|ool readiness<br>on sample|2 years of school<br>readiness<br>intervention<br>sample|
|---|---|---|---|
||end of 1st grade<br>average TEGRA|end of 2nd grade<br>average TEGRA|end of 1st grade<br>average TEGRA|
||score|score|score|
|girls||||
|in a CLRW control school|0.17|0.3**|0.29**|
||(0.15)|(0.12)|(0.13)|
|in a CLRW treatment school|0.18|0.09|‐0.09|
||(0.14)|(0.12)|(0.11)|
|difference|0.01|‐0.21|‐0.38**|
||(0.21)|(0.17)|(0.17)|
|boys||||
|in a CLRW control school|0.12|0.28*|0.01|
||(0.13)|(0.15)|(0.1)|
|in a CLRW treatment school|0.08|0.1|‐0.09|
||(0.13)|(0.14)|(0.08)|
|difference|‐0.04|‐0.17|‐0.1|
||(0.18)|(0.21)|(0.12)|
|sample size|1049|925|1140|

Standard errors included in parentheses. Statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively. Effects are estimated by equation (9), presented in Annex Table 9.

# _Discussion_

The school readiness intervention positively affected student reading outcomes for girls in the CLRW control group and, in one case, boys. This reflects the findings in Table 4 showing that girls benefited more broadly than boys from the intervention. The effect size for girls in a CLRW control school exposed to two years of the school readiness intervention and one year of schooling is 0.3 standard deviation, and it is higher than the effect size for girls exposed to one year of the school readiness intervention and one year of schooling. This suggests that the 39
school readiness intervention enhanced the effect of a year of schooling, and this enhancement is consistent with how the school readiness intervention was expected to affect reading outcomes. Exposure to the school readiness intervention intended to improve the home environment and subsequently a child’s preparedness to learn in school.

Table 15. Comparison of effect per US$ 100

|school readiness|CLRW|difference|
|---|---|---|
|intervention|||
|_1 year of school readiness intervention samp_<br>effect on end of 1st grade reading skills<br> <br>|_le_<br>||
|females<br>1.37|0.41***|0.96|
|(1.2)|(0.13)|(1.14)|
|males<br>0.98|0.44***|0.54|
|(1)|(0.12)|(0.95)|
|effect on end of 2nd grade reading skills|||
|females<br>2.36**|0.28***|2.08**|
|(0.96)|(0.08)|(0.93)|
|males<br>2.19*|0.37***|1.82|
|(1.22)|(0.09)|(1.17)|
|_2 years of school readiness intervention sam_<br>effect on end of 1st grade reading skills|_ple_||
|females<br>1.17**|0.43***|0.74|
|(0.53)|(0.12)|(0.48)|
|males<br>0.04|0.3***|‐0.26|
|(0.38)|(0.1)|(0.35)|

Standard errors included in parentheses. Statistical significance at the 1, 5 and 10 percent levels are denoted by ***, **, and *, respectively. Effects are relative to being exposed to neither intervention and calculated by dividing the effects presented in Annex Table 9 by the per student costs in 100s.

No complementarity was found between the school readiness intervention and the CLRW intervention except among girls with two years of exposure to the school readiness intervention; for this sub-population, the effect of the school readiness intervention was found to be smaller in CRLW treatment schools than in control schools. For all other sub-populations, no statistically significant difference in the impact of the school readiness intervention between CLRW 40
treatment and control schools was found. The school readiness intervention was less costly than the CLRW intervention; however, only for one-subpopulation can we conclude that the school readiness intervention was more cost effective.

To estimate the effect of the school readiness intervention on reading scores, we relied on a mapping of students between communities and schools defined at the school level. This provided a probability that a student in a particular school was from a school readiness treatment community. The resulting estimator is unbiased under several conditions. First, if the proportion of students in a school from a treatment community is uncorrelated with the unobserved school effect. This is a reasonable assumption because the assignment of communities was randomized. Second, the probability of a student being from a treatment community is estimated based on a sub-sample of students matched between the TEHCI and TEGRA surveys. Uncertainty of this estimate creates the well-known error-in-regressors bias. If it acts as attenuation bias as in a univariate regression model, then this results in an underestimate of the effect size.

# <u>6. Conclusion</u>

This study evaluates two quite different approaches to improve early educational outcomes in Tonga. The school readiness intervention targets the home environment and was implemented by communities; the CLRW intervention targets schools and was implemented more directly by the education ministry. Positive effects for the school readiness intervention were found on children of mothers without a high-school education on the overall school readiness score for girls and the literacy score for both boys and girls of the same mothers. It also affected some specific measures of home activities as well. The CLRW positively affected observed teaching practices in the classroom as well as reading skills for both boys and girls and most domains of reading skill. These findings suggest that expansion of both the CLRW and school readiness interventions is warranted.

Because two cohorts were potentially exposed to both interventions, we are able to compare the effects of these two very different interventions on common measures of learning outcomes for 41
the same population of children. The school readiness intervention had a positive effect on early grade reading skills but only for select sub-populations and only those in CLRW control schools. No complementarity was found between the two interventions, and for one sub-population, the CLRW intervention reduced the effect of the school readiness intervention. While we find that the school readiness intervention was no less cost effective than the CLRW intervention for improving reading scores, we do not conclude that the school readiness intervention is necessarily an alternative to the CLRW intervention for improving reading outcomes specifically. Rather, comparing these two interventions’ effects on the same population of children and outcomes clearly demonstrates the potential of a school readiness intervention, implemented by communities and acting through a complex and indirect chain—via community participation, parental awareness, parenting and the home environment, school readiness and finally learning outcomes—to be among the options for education policy makers to improve learning outcomes more generally.

42