metadata: authors: 0: author: Delavallade,Clara Anne
1: author: Griffith,Alan
2: author: Thornton,Rebecca Lynn
count: India
docna: 0: docna: Effects of a Multi-Faceted Education Program
            on Enrollment, Equity, Learning, and School Management :
            Evidence from India
docty: Policy Research Working Paper
subtopic: Educational Sciences,Educational Institutions & Facilities,Gender and Development,Effective Schools and Teachers,Primary Education
teratopic: Gender,Education
abstracts: cdata!: The Sustainable Development Goals set a
            triple educational objective: improve access to, quality of,
            and gender equity in education. This paper documents the
            effectiveness of a multifaceted educational program,
            pursuing these three objectives simultaneously, in rural
            India. Using an experiment in 230 schools, the paper
            measures the effects of the program on students' school
            participation and academic performance over two years, while
            also examining heterogeneous impacts and sustainability. The
            findings show that the program increased enrollment,
            especially among girls (8.1 percent in the first year, 11.7
            percent in the second), reducing gender gaps in school
            retention. The findings show large learning gains of 0.323
            standard deviation due to the program in the first year and
            0.156 standard deviation at the end of the second year,
            which did not vary by gender. There were also large effects
            on school management outcomes, increasing the number of
            meetings by 16 percent and the number of improvement plans
            completed by 25 percent.
geo_reg_key: 517198,517191,517212
repnme: repnme: Effects of a Multi-Faceted Education
            Program on Enrollment, Equity, Learning, and School
            Management : Evidence from India
display_title: Effects of a Multi-Faceted Education
            Program on Enrollment, Equity, Learning, and School
            Management : Evidence from India
listing_relative_url: /research/2019/12/31633097/effects-multi-faceted-education-program-enrollment-equity-learning-school-management-evidence-india
topicv3: Teachers Management m1326119 1523,Primary Education m1326139 480,Education m1327048 470,Gender m1327621 615,School-Based Management m1327730 2143
publishtoextweb_dt: 2019-12-12T00:00:00Z
historic_topic: Gender,Education
origu: Gender Director (HGNDR)
no_of_pages: 46
author_keys: 0: author_key: 1594950
1: author_key: 684219
model_extractions: Policy Research Working Paper 9081 Effects of a Multi-Faceted Education Program on Enrollment, Equity, Learning, and School Management

# Evidence from India

_Clara Delavallade Alan Griffith Rebecca Thornton_ Africa Region Gender Innovation Lab December 2019
Policy Research Working Paper 9081

# **Abstract**

The Sustainable Development Goals set a triple educational objective: improve access to, quality of, and gender equity in education. This paper documents the effectiveness of a multifaceted educational program, pursuing these three objectives simultaneously, in rural India. Using an experiment in 230 schools, the paper measures the effects of the program on students’ school participation and academic performance over two years, while also examining heterogeneous impacts and sustainability. The findings show that the program increased enrollment, especially among girls (8.1 percent in the first year, 11.7 percent in the second), reducing gender gaps in school retention. The findings show large learning gains of 0.323 standard deviation due to the program in the first year and 0.156 standard deviation at the end of the second year, which did not vary by gender. There were also large effects on school management outcomes, increasing the number of meetings by 16 percent and the number of improvement plans completed by 25 percent.

This paper is a product of the Gender Innovation Lab, Africa Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted atcdelavallade@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Effects of a Multi-Faceted Education Program on Enrollment, Equity, Learning, and School Management: Evidence from India**

Clara Delavallade, The World Bank Alan Griffith, University of Washington Rebecca Thornton, University of Illinois<sup>1</sup> JEL classification: O15, I25, J16, O19 Keywords: education, randomized experiment, school effectiveness, learning, gender equality 1 Delavallade: Africa Gender Innovation Lab, World Bank, 1818 H Street, NW Washington, DC 20433, US (email: cdelavallade@worldbank.org), Griffith: Department of Economics, University of Washington, Savery Hall, 410 Spokane Ln, Seattle, WA 98105, US (email: alangrif@uw.edu), Thornton: Department of Economics, University of Illinois at Urbana-Champaign, 214 David Kinley Hall, 1407 W. Gregory, Urbana, IL 61801, US (email: rebeccat@illinois.edu). The authors thank the members of the NGO field team for their cooperation and contributions in all stages of the project and would especially like to acknowledge the useful comments provided by many seminar participants. Oxana Azgaldova, Kwong-Yu Wong, and Flor Paz provided valuable research assistance. All errors are our own.
# **1. Introduction**

The United Nations’ Sustainable Development Goal 4 proposes that by 2030 “all girls and boys [should] complete free, equitable, and quality primary and secondary education” (United Nations 2015). This ambitious goal sets a triple objective for educational policies: improving free access to, equity in, and quality of learning. It is unlikely that one single intervention is enough to make headway on these objectives simultaneously. For example, policies that are successful in increasing access to school may not necessarily foster quality education, and interventions that are effective among girls, may not be equally effective for boys. However, while education programs implemented by governments or NGOs often involve multiple inputs and programmatic objectives, we know very little about their effectiveness. The vast majority of education evaluations study programs that focus on only one objective or educational input – for example, only 5 percent of the studies in McEwan’s 2015 meta-analysis of primary school evaluations measure the impact of a “combination” treatment.

In contrast, this paper presents the results from an evaluation of a multifaceted educational program designed and implemented by an India-based nongovernmental organization (NGO) in rural Rajasthan, India. The program had three primary objectives: to enroll marginalized girls, improve student learning, and support school management. The program involved multiple interventions: door-to-door campaigns to enroll drop-out or never-enrolled girls, volunteers trained to teach activity-based and playful learning among students grouped by ability, strengthening school management committees, and working with members of the community to promote girls’ education. The multiple inputs were designed to complement and support the objectives of the program. By grouping students according to ability, the volunteer-led teaching activities could help teachers cater to students of varying abilities and mitigate potential harmful effects from the enrollment drive. Similarly, the program’s community engagement and sensitization to the 2
importance of girls’ education could mitigate potential negative effects of targeting girls for enrollment, rather than focusing on both genders.

To evaluate this program, we use a cluster-randomized experiment in 230 primary schools and individual-level panel data on enrollment, retention, attendance, and test scores in English, Hindi, and Math. We examine whether the program met its objectives: improving enrollment – especially of girls, increasing learning – across all abilities, and supporting school management committees. To measure whether the program was successful at targeting – to girls and students of lower-ability – we examine how the program differentially affected girls (vs. boys), and initially low-performing students (vs. higher performing). We also discuss threats that may arise from differential retention and enrollment.

We find large and statistically significant positive impacts on student enrollment – especially among girls – in the two years of the program. Further, these effects are large, representing increases of 8.1% and 11.7% in the first and second year of the program, respectively. While estimated effects are larger for girls, the differences between girls and boys are not statistically significant. Further, we find no statistically significant effects on the types of students (i.e., high vs. low ability) who enroll in program schools.

In terms of learning, we find large positive gains in all subjects, among both boys and girls. In the first year of the program, the impacts on post-program test scores are highest: 0.323 standard deviation across all subjects (0.317 in Hindi, 0.256 in English, and 0.369 in Math). Students in treatment schools are also 22.7 percentage points more likely to improve their scores in the first year when comparing pre-program with post-program tests. There are no significant differences in the effect of the program on learning gains by student gender. Second year post-program impacts are similar, although somewhat smaller, at 0.156 SD (0.127 in Hindi, 0.159 in English, and 0.136 3
in Math), but the difference between the post-program tests across the two years is not statistically significant (p=0.226).

Tests were also administered early at the beginning of the school year of the second year of the program – after the first year of programming, but prior to program implementation in the second year. We find significantly lower learning gains on this test among students who were exposed to the program in the prior year, suggesting that either the program resulted in teachers “teaching to the test” (Glewwe, Ilias and Kremer 2010), or in a loss in learning gains over school breaks, similar to summer vacation losses in the United States (Cooper et al., 1996).

We also examine how the intervention affected learning across students of varying initial academic performance and find that, at least in the first year of the program, gains increased as one moves upward in the ability distribution. This is similar to the findings in Bulh-Wiggers et al. (2019b), who also find larger effects of an education program among those of highest ability. This speaks to a related literature that finds larger sources of inequality in learning across dimensions other than gender – for example, across ability, or wealth (Crouch and Rolleston 2016, Kaffenberger and Pritchett 2017).

Lastly, we find that the program led to more School Management Committee meetings, as well as a larger number of prepared and completed improvement plans. In particular, the program increased the number of meetings held by 16%, the number of prepared improvement plans by 22% and the number of completed improvement plans by 25%.

The effects we find on learning are about twice as large as evaluations of other learningtargeted interventions that focus on one single educational input, such as using contract or volunteer teachers (average effect size = 0.10SDs) or training teachers (average effect size = 0.12SDs; McEwan 2015).

4
Our results speak to the effectiveness of multi-pronged interventions that aim to improve access without sacrificing quality of education. Many education programs implemented in developing countries involve a combination of interventions and target multiple issues at the same time. For example, among all registered voluntary and non-governmental organizations in India, only 16% that self-identify as working within the Education and Literacy sector operate exclusively within that sector.<sup>2</sup> Of those operating exclusively within the Education and Literacy sector, very few operated using a single programmatic strategy.<sup>3</sup> Similarly, of the four main federal “School Education & Literacy Schemes” for primary education in India, only one, the Mid- day Meal Program, focuses on one single intervention.<sup>4</sup> In contrast to the breadth of activities, approaches, and objectives that are commonly implemented to improve education in developing countries, a majority of the studies of education programs involve measuring the effect of one type of intervention in isolation. McEwan (2015) studied 76 RCTs and finds only 5 percent evaluate a “combination” of treatments (authors’ calculations).<sup>5</sup> In addition to RCTs, “high quality” studies involving regression discontinuity (RD) and difference in difference (DD) evaluations also tend to focus exclusively on programs with one intervention implemented in isolation. Damon et al. (2016) reviewed 39 “high quality” RD and

> 2 Authors’ Calculations. We extracted the list of NGOs/VOs through NGO-DARPAN at <u>https://ngodarpan.gov.in</u> Registered VOs and NGOs are required to sign-up online using the portal, run by the Planning Commission of India (PTI 2017). Accessed 12/3/2018.

> 3 After coding all NGOs in the state of Rajasthan that focused exclusively on Education and Literacy we found 2.3% of NGOs focused on research or academia, 4.1% focused on giving financial support to students, 56.1% involved running a school or set of schools, 14.62% provided vocational training to youth or adults, 18.7% were coded as providing multiple types of education programs, and 4.1% were not known.

> 4 See http://mhrd.gov.in/schemes-school.

> 5 McEwan’s literature search of randomized experiments conducted in developing country primary schools from the mid-1970s to 2013 resulted in the following studies that evaluated multiple treatment: He, Linden, and MacLeod (2008), Osendarp et al. 2007, Pradhan, Suryadarma, and Beatty (2011), and Ngyuen (2008).

5
DD studies. We coded each study to find that only 10.2 percent evaluate a combination of interventions.<sup>6</sup> There are a large number of papers that have studied, in isolation, the specific interventions that are implemented in the program we evaluate. First, while we are unaware of any study that evaluates the effectiveness of an “enrollment drive,” a sizable literature has evaluated various approaches to improve student enrollment (See JPAL Policy Bulletin 2017). Second, the activitybased learning among students grouped by ability is similar to “teach at the right level” or targeted interventions, that have seen overwhelming success across several settings (Banerjee, Cole and Duflo 2007, Banerjee et al 2010, Banerjee et al. 2016, Duflo, Dupas and Kremer 2011, Muralidharan Singh and Ganimian, 2018). Third, the use of volunteers to deliver their programmatic activities has also been evaluated, with mixed evidence (Banerjee, Cole, Duflo, and Linden 2007, Lakshminarayana et al. (2013), Banerjee et al. 2010, Torgerson, King and Sowden 2002). Lastly, the NGO’s focus on school management and community engagement is similar to interventions that have evaluated community management or parent involvement (Barr et al. 2012, Lassibille et al. 2010, Gertler, Patrinos and Rodriguez-Oreggia 2012, Beasley and Huillery 2017, Banerjee et al 2010, Proadhan et al. 2014, and Blimbo, Evans and Lahire 2015, Glewwe and Maïga 2011).

This paper contributes to a number of additional literatures. For example, there is a growing body of evidence on the effectiveness of girl-focused interventions to address the gender gap in schooling. Evans and Yuan (2018) provide a review of these interventions but find that

> 6 The papers contained in Damon et al. (2016) consist of 115 studies – 76 RCTs and 38 “High-Quality” regression discontinuity or difference in difference studies. The search involved studies conducted from 1990 to 2014, published in (peer-reviewed) academic journals from 1990 to 2014, and unpublished academic working papers written from 2010 to 2014 were included. Papers with multiple interventions include: Chey, McEwan, and Urquiola (2005), Gertler, Patrinos, and Rubino-Codina (2012), Kazinga et al. (2013), Santibanez, Abreu-Lastra, and O’Donoghue (2014).

6
interventions targeting girls result in no real advantage over education programs targeting both boys and girls.<sup>7</sup> While we find some differences in enrollment and retention gains among girls, there were no differences in the effect of the program on learning by gender. Kremer, Miguel, and Thornton (2009) found that boys benefited from a merit-based scholarship program even if they were not eligible themselves. In contrast, boys responded negatively to being excluded from a gender-based life skills program (Delavallade, Griffith, and Thornton, 2016).

Finally, our paper adds to the literature on the sustainability of treatment effects over time (Banerjee et al. 2007; Kremer and Miguel 2007; Andrabi, Das, and Khwaja 2008; Kremer, Miguel, and Thornton 2009; Duflo, Dupas, and Kremer 2011; Baird et al. 2016). Most evaluations of education programs conduct just one follow-up after the intervention to measure impact: McEwan (2015) found that the average number of follow-ups per experiment was just 1.4. Our multiple rounds of follow-up data help document some fade-out of the program effects on post-program tests in the second year, as well as document large declines in the learning gains on pre-program tests in the second year. These findings suggest there may be learning losses between academic years similar to the literature on summer vacation loss found in the United States (Cooper et al., 1996). We next present the background of the setting and education intervention in Section 2. Section 3 presents the research design, Section 4 presents the results, Section 5 discusses and presents additional robustness and Section 6 concludes.

> 7 Some examples of particularly successful strategies for girls include programs that reduce the direct costs of schooling (Bruns, Mingat, and Rakatomalala 2003; Deininger 2003; Muralidharan and Prakash 2017), reduce the indirect costs and opportunity costs (Khandker, Pitt, and Fuwa 2003; Lavinas 2001), involve communities (Herz 2002; Benveniste and McEwan 2000), make schools girl-friendly (World Bank 2001; Herz 2002), and improve the quality of education (Lloyd, Mensch, and Clark 1998; Khandker 1996).

7
# **2. Background**

# **2.1 Schooling in Rajasthan: Participation, Quality, and Gender Equity**

Despite educational advances in most developing countries, the state of Rajasthan in India has experienced limited educational gains over the past decade, especially for girls (World Bank 2011). In 2012, 4.6 percent of girls 7–10 years old were still not in school in rural Rajasthan, compared with 2.2 percent of boys (Pratham Organization 2012). This gender gap widens considerably as students age, due largely to social norms that particularly disadvantage girls. Marriage is often seen as a substitute for schooling, and girls frequently have little say in when and whom they marry. In addition, marriage often occurs at a young age, with 57.6 percent of women marrying younger than the legal age (UNICEF 2012, 173).

In addition, educational quality is low, with only 47.7 percent of children in grades three to five able to read a grade one–level text in government schools in 2011, and only 33.1 percent able to do subtraction in 2012 (Pratham Organization 2012). The availability of primary schools in remote areas is still limited, leading to high variance in student-teacher ratios.

# **2.2 The Intervention**

We evaluate an intervention developed and implemented by an Indian NGO working with government schools in the state of Rajasthan. One of its main aims is to increase girls’ educational outcomes, with a focus on increasing school participation and learning in lower primary school (grades one through five). The program consists of several components that separately target enrollment and retention, learning, and school management. Each of the components of the program is directed by a trained volunteer in each village.

8
To target enrollment, the NGO identifies out-of-school girls before each school year, using information from community members and government records. The program volunteers hold village meetings to prepare for a house-to-house enrollment drive, targeting girls who have never enrolled or who have dropped out of school. These efforts seek both to encourage parents to support their daughters’ education and to motivate girls themselves to come to school.<sup>8</sup> To target student learning, the NGO organizes in-school lessons led by the volunteer in grades three through five. The curriculum and instructional model was designed with Pratham Rajasthan and emphasizes activity-based and playful learning through games that teach English, Hindi, and Math. The methodology emphasizes group work and student involvement in the teaching and learning process. These lessons are held during school hours for approximately two hours per day, several days per week, over four to five months. This component of the program does not focus explicitly on girls, but rather aims to increase learning levels for both girls and boys.<sup>9</sup> In tandem with the peer group learning method, students are placed in three groups according to ability, measured by diagnostic pre-program tests similar to the Annual Survey of Education Report (ASER); the tests are designed to be quick to administer so that they can be conducted individually for each student.

The program was implemented and evaluated in the academic years of 2012 and 2013. Each year, in selected villages, village volunteers conducted the door-to-door enrollment drive,

> 8 The NGO identifies out-of-school girls (aged 6 to 14) using a two-step approach. Using data from the state government child tracking system (CTS) and school records, the program develops an initial list of girls to target. To prepare for the door-to-door enrollment drive, the program then engages community members to verify records, build awareness, and increase enrollment. Volunteers organize meetings of 20 to 40 individuals, to engage village leaders and community members as champions for increasing girls’ school enrollment, and jointly identify barriers and develop solutions. Volunteers also leverage local meetings with other programs.

> 9 The program involves a 12-week module focused on interactive teaching methods, “Catch Up” methodology for children who are behind their grade level, and peer group learning. After training, the program provides teachers with learning kits, which includes games and creative learning materials, and access to a telephone helpline and SMS updates for extra support. In schools with poorer performance, the NGO provides additional “handholding” visits to support teachers and the trained volunteer.

9
carried out the learning curriculum in grades three, four, and five, and assisted with strengthening School Management Committees and community support for girls’ education. Appendix A presents additional details about the intervention and implementation.

# **3. Research Design**

# **3.1 School Sample and Randomization**

The study consists of 230 primary schools located in 98 villages in Rajasthan. In four administrative blocks, villages with at least one government primary school were selected for the study. In 2011, prior to the implementation of the program, researchers randomly assigned villages to either treatment or control groups (49 treatment and 49 control), stratified by Administrative Block, using a random number generator. This results in 117 treatment schools and 113 control schools. On average, there are 2.3 government primary schools per village.

# **3.2 Data and Outcome Measures**

# _Baseline Data_

We use data collected in 2011 – prior to the implementation of the program – to check for baseline balance and as covariates in the analyses. At the school level, we use school infrastructure data (such as the presence of electricity and computers and the number of students at each school) collected by the NGO staff. At the individual level, enrollment rosters collected in 2011—prior to program implementation—list each student’s gender, grade, age and whether he or she belongs to a Scheduled Caste, Scheduled Tribe, or Other Backward Caste. We also construct school-gradelevel enrollment for boys and girls in 2011 that we use as controls.

10
_Learning_ Learning is measured using tests conducted in school in 2012 and 2013, conditional on the student being enrolled and present on the day of the exam. In each year, two exams were administered: a pre-program test conducted just prior to, and a post-program test conducted just after, the learning component of the program was implemented.<sup>10</sup> The tests were based on ASER exams, which are standardized exams validated across India testing Hindi, English, and Math.<sup>11</sup> The exams are short, taking 5-10 minutes per student, testing the same skills in both years. For Hindi and English, students are tested on letters, words, a short paragraph, and a longer story, while math tests knowledge of single- and double- digit number recognition, two-digit subtraction with borrowing, and three-digit by one-digit division. Enumerators assess the highest level a student can comfortably perform. Following the ASER criteria, the tests are scored categorically from A (highest score) to E (lowest).

We construct two measures of learning from the exam scores. First, we normalized test scores to the control after assigning each letter grade a numeric value from one (E, lowest) to five (A, highest). Second, we create an indicator for whether the student’s test score improved from pre-treatment to post-treatment. These two measures are motivated by the observation that, with the raw scores on a categorical scale, the difference between two adjacent scores may not be constant in terms of measuring learning outcomes. The indicator for a student’s score increasing represents a way of looking at non-linear effects.

Tests were administered conditional on the student being present in school on the day of the exam. We address the possible bias due to differential test-taking in our analysis.

> 10 In 2012, pre-program tests were administered in Nov-Dec 2012, while post-program tests were administered in March 2013. In 2013, pre-program tests were administered in Sept-Oct 2013, while post-program tests were administered in March 2014.

> 11 See www.asercentre.org for more information about the ASER tool.

11
# _Enrollment_

To measure enrollment, we use school enrollment rosters collected each year. From these data, we also construct school-grade-level total enrollment, by gender and pooled across girls and boys.

# _School Management_

To understand the impact of the program on school management outcomes, we collected the number of School Management Committee (SMC) meetings held, and the number of School Improvement Plans (SIPs) prepared and implemented at each school. These were collected monthly from July 2012 to January 2013, for a total of seven observations per school.

# **3.3 Baseline Sample Students and Pre-Program Balance**

Students who were enrolled in grades three and four at the beginning of 2011 (prior to the intervention) comprise our _Baseline Sample_ (N=7,327). Table 1 presents average statistics and balance tests of pre-program student and school-level characteristics. On average, 55 percent of the students are girls, with the majority of students belonging to a scheduled caste (31 percent), scheduled tribe (15 percent) or other backward caste (40 percent). Less than half – 38 percent – of the students are in schools with electricity, few – 10 percent – have access to a computer, and just over 80 percent have access to clean drinking water.

Column 2 presents the regression coefficient testing the difference in means of pre-program variables between the treatment and control groups. We see no significant differences between treatment and control students across gender, caste, or school characteristics such as type of school (upper or lower secondary), having electricity, a computer, or drinking water, although students in treatment schools are younger on average. There are slightly fewer students enrolled in treatment 12
|Table 1 — Pre|-program Chara|cteristics ofStu|dents: Ba|seline Sample|||
|---|---|---|---|---|---|---|
|||All||Gris||Boys|
||Meanin<br>Control|Difference<br>Treat-Control|Meanin<br>Control|Difference Treat- <br>Control|Meanin<br>Control|Difference Treat -<br>Control|
||Q)|(2)|(3)|(4)|(3)|(6)|
|Girl|0.545|-0.016|||||
||(0.014)|(0.019)|||||
|Grade 4 at Baseline|0.332|-0.009|0.334|0.006|0.331|-0.022|
||(0.008)|(0.012)|(0.010)|(0.016)|(0.011)|(0.015)|
|Age|10.070|-0.324**|10.003|-0.260**|10.127|-O.377***|
||(0.066)|(0.095)|(0.076)|(0.105)|(0.081)|(0.110)|
|Scheduled Caste|0.305<br>|-0.014<br>|0.323<br>|-0.028<br>|0.289<br>|-0.002<br>|
||(0.030)|(0.038)|(0.034)|(0.045)|(0.029)|(0.036)|
|Scheduled Tribe|0.130<br>|0.010<br>|0.169<br>|0.029<br>|0.135<br>|-0.008<br>|
||(0.022)|(0.036)|(0.027)|(0.046)|(0.022)|(0.032)|
|Other Backward Caste|0.400<br>|-0.015<br>|0.394<br>|-0.041<br>|0.405<br>|0.008<br>|
||(0.034)|(0.052)|(0.040)|(0.059)|(0.032)|(0.050)|
|UpperPrimary School (UPS)|0.582|-0.022|0.567|-0.008|0.594|-0.035|
||(0.073)|(0.090)|(0.078)|(0.096)|(0.072)|(0.089)|
|Baseline Students Enrolled in Same Grade|23.003<br>|-1.474<br>|23.293<br>|-1.653<br>|22.762<br>|-1.331<br>|
||(1.817)|(2.310)|(1.890)|(2371)|(1.808)|(2.327)|
|Baseline Students Enrolled in AllGrades|64.985<br>|-4.913<br>|66.066<br>|-5.639<br>|64.083<br>|-4.328<br>|
||(5.318)|(6.575)|(5.633)|(6.843)|(5.174)|(6.509)|
|Electricity|0.396<br>|0.014<br>|0.386<br>|0.026<br>|0.404<br>|0.005<br>|
||(0.066)|(0.095)|(0.065)|(0.097)|(0.069)|(0.096)|
|Computer|0.117<br>|-0.023<br>|0.107<br>|-0.020<br>|0.126<br>|-0.026<br>|
||(0.041)|(0.054)|(0.040)|(0.054)|(0.044)|(0.057)|
|DrinkingWater|0.806<br>|0.022<br>|0.803<br>|0.017<br>|0.808<br>|0.027<br>|
||(0.059)|(0.076)|(0.065)|(0.082)|(0.057)|(0.074)|
|Teacher/Pupil Ratio|0.069<br>|-0.002<br>|0.070<br>|-0.004<br>|0.068<br>|0.000<br>|
||(0.007)|(0.008)|(0.007)|(0.008)|(0.007)|(0.009)|

‘Notes: Robust standard errors in parentheses, clustered by village. For differences, *** p<0.01, ** p<0.05, * p<0.1. Baseline Sample consists ofstudents enrolled in 2011 in grades 3 and 4. N=7,327 students in 230 schools.
effect of the program on learning and explore whether the program had differential effects on girls and low-ability students. Lastly, we estimate effects of the program on school management outcomes.

# _Effect of the Program on Enrollment_

To measure the effects of the program on enrollment, we test the difference in average grade-level enrollment by estimating the following: (1) Ygsj � β0 � β1Tj � γ<sup>′</sup> Xgsj � εgsj where Ygsj identifies the number of students enrolled in grade _g_ in school _s_ in village _j_ , in either 2012 or 2013. Tjindicates whether village _j_ was assigned to the treatment group. The vector Xgsj is included in some specifications for statistical power and includes the number enrolled by gender prior to program implementation and cohort fixed effects. We estimate the equation with a linear probability model and cluster standard errors by village, the unit of randomization. We estimate the equation for all students and separately by gender.

# _Selection into Test-Taking_

Before estimating the effect of the program on learning outcomes, we examine whether the program affected the type of students we observe for learning outcomes. Since test scores are only observed conditional on enrollment and attendance on the day of the exam, if the program was successful at targeting low-performing students, the estimates of the program’s impact on test scores would be biased downward. Conversely, estimates may be biased upward if higherperforming students were more likely to be retained or attend on the test day in treatment schools. To address potential threats due to differential test-taking, we examine the effects of the 14
program on pre-program and post-program test-taking in 2012 and 2013, as well on the likelihood of taking both tests, with Equation (1). In this context, Ygsj is the number of students who were present and took the test. Using demographic data and pre-program test scores, we also characterize the types of students who are test-takers in the treatment and control groups.

_Effect of the Program on Learning_ Our sample for the analysis on learning outcomes includes students who took both preprogram and post-program tests in either 2012 or 2013. We estimate the effect of the program on exams in 2012 and 2013, pooling all subjects—Hindi, English, and Math—with the following regression: (2) Yisjz � β0 � β1Tj � γ<sup>′</sup> Xisj � εisjz for individual _i_ , in school _s_ , village _j_ , on subject _z._ Learning, Yisjz, is measured with either normalized test scores or with an indicator for whether the student’s test score improved from pretreatment to post-treatment in a given year. We include subject and cohort fixed effects as well as controls for school size prior to program implementation, captured by γ<sup>′</sup> Xisj in the above equation. We cluster standard errors by village.

We run Equation (2) separately for pre-program and post-program test scores in each of 2012 and 2013. Recall that the pre-program test in 2012 was administered prior to the implementation of the learning component of the program and we would not expect any learning gains on this test. Pre-program tests in 2013 were administered prior to program implementation in 2013, but students in treatment schools who were enrolled in 2012 would have had exposure to the program. The results on 2013 pre-program tests indicate whether any learning gains experienced in year one were sustained into the beginning of year two.

15
<!-- Start of picture text --> =B +B +B +B xGirl +y +e<br>= Dest (B +B ) +yX +e<br>=B xT +B xT +B xT+<br>+€<br>=) ,=11{Pretest =v}(B +5 )+yxX +e<br><!-- End of picture text -->
where Pretestijs indicates the pre-program test score for student _i_ in village _j_ on subject _s_ . Differences in the coefficients δv identify different treatment effects by pre-program test score, while differences in the coefficients βv indicate different means among students in control schools by pre-program score. We include cohort and subject fixed effects in all specifications as well as controls for school size at baseline.

We estimate Equations (4), (5), and (6), for all students pooled together as well as boys and girls separately. All specifications include three outcomes per student for each test administration, because students took tests in three subjects.

# _Effects on School Management Committee Outcomes_

Lastly, since one aim of the program is to build capacity through School Management Committees (SMCs), we measure the program’s effect on indicators of SMC activity. To do this, we compare the average number of SMC meetings, and number of improvement plans prepared and completed among treatment and control schools. In this manner, we test whether the program led to more SMC activity.

# **4. Effect of the Program on Enrollment, Learning and School Management**

# **4.1 Enrollment**

Table 2 shows the estimates of the program’s impact on the number of enrolled students in 2012 and 2013. Columns 1-2 show results for all students, while Columns 3-6 disaggregate the results by gender. In 2012 – the first year of the program, the program increased the number of students per grade by 0.655 (Panel A, Column 2). There is no detectable effect on boys (Columns 3-4), while the effect on girls is large and statistically significant (Column 6). The program effect on girls amounts to an increase of 0.684 additional students, representing an increase of 8.1% from 17
|iT able <br>Panel A: 2012<br>|2 --Treatment <br>|Effects on Stu<br>|dent Enrollmen<br>|t: School-Gra<br>|deLevel||
|---|---|---|---|---|---|---|
|Population Group|All St|udents|Bo|ys||Girls|
|Treatment|(1)|(2)|(3)|(4)|(5)|(6)|
||0.162|0.655|-0.043|-0.030|0.206|0.684**|
||(1.302)|(0.539)|(0.716)|(0.303)|(0.737)|(0.336)|
|Controls|NO|YES|NO|YES|NO|YES|
|Observations|690|690|690|690|690|690|
|R-squared|0.000|0.694|0.000|0.653|0.000|0.633|
|Meanofdepvarincontrol|15.643|15.643|7.188|7.188|8.455|8.455|

|Panel B: 2013<br>Population Group|All St|udents|Bo|ys|Gi|rls|
|---|---|---|---|---|---|---|
|Treatment|(1)|(2)|(3)|(4)|(5)|(6)|
||0.801|1.286*|0.352|0.399|0.450|0.887**|
||(1.276)|(0.682)|(0.696)|(0.392)|(0.716)|(0.395)|
|Controls|NO|YES|NO|YES|NO|YES|
|Observations|690|690|690|690|690|690|
|R-squared|0.001|0.583|0.001|0.525|0.001|0.532|
|Meanofdepvarincontrol|14.185|14.185|6.586|6.586|7.598|7.598|

‘Notes: Robust standard errors in parentheses, clustered by village. *** p<0.01, ** p<0.05, * p<0.1. The sample includes students in grades 3,4, and 5 in 230 schools. Observations are at the school-grade-level. Controls include 2011 (pre-program) enrollment by gender and cohort fixed effects where indicated, in Columns 2, 4, and 6.
# **4.2 Program Effects on the Composition of Test-Takers**

Before turning to the results on learning, we first examine the potential for bias due to differential test-taking caused by the program. If more marginal students were more likely to take the learning assessments in treatment schools, our program effect estimates might be biased downward.

Appendix Table A1 presents the effect of the program on school-grade-level test-taking in 2012 and 2013. We present the effect of the program on taking each of the pre- and post-program tests, as well as for taking both tests. While noting the relatively large standard errors, we see no significant effects of the program on test-taking in 2012 (Panel A) – despite the effects on enrollment found in Panel A of Table 2. This result suggests that the program could enroll students, but not necessarily increase attendance or test-taking.

We do find significant effects on the number of students who take tests in 2013 (Panel B). In 2013, there were 0.6 more girls and 0.3 more boys in program schools who took the pre-program test (Columns 2 and 3). This is consistent with the increase in student enrollment in 2013 (Panel B, Table 2). Girls were also more likely to take the post-program tests (Columns 6 and 9).

Next, we examine what types of students are more likely to be test-takers across the treatment and control in Appendix Table A2. This analysis is restricted to our test-taker sample, defined as those who took both tests in 2012 (Columns 1-2) or 2013 (Columns 3-4). Since we have pre-program data for all these students, we can analyze whether students in treatment schools were weaker on average. We find no evidence of this. Test-takers in treatment schools were marginally more likely to be girls, but we see very little evidence that their test scores were lower. We interpret this as a lack of evidence that the program caused differential test-taking, suggesting limited scope for bias in our analysis of effects on learning.

19
Table 3 — Treatment Effects on Learning Outcomes

|Panel A: 2012<br>|||||||
|---|---|---|---|---|---|---|
|Outcome:||Normalized|TestScores||Increased|rompre- to|
||Pre-pro|<br>gramtest|<br>Post-pro|gram test|<br>post-pro|<br>gram test*|
||(1)|(2)|(3)|(4)|(5)|(6)|
|Treatment|0.007|0.009|O.325***|O321**|QaaI7e|=O.250F**|
||(0.090)|(0.096)|(0.094)|(0.097)|(0.059)|<br>(0.061)|
|Girl||-O.117***||-0.084**||0.034**|
|||(0.035)||(0.035)||(0.014)|
|Treatment < Girl||-0.008||-0.012||-0.017|
|||(0.056)||(0.056)||(0.020)|
|Controls|NO|YES|NO|YES|NO|YES|
|Observations|17,460|17,460|17,460|17,460|16,347|16,347|
|R-squared|0.000|0.173|0.024|0.233|0.055|0.067|
|Meanofdepvarincontrol|0.000|0.000|0.000|0.000|0.512|0.512|
|P-value:<br>T+ T x Girl=0||0.991||0.002||0.000|
|Panel<br>|||||||
|Bi2013.||Normalized|TestScores||Increased|rompre-to|
|<br>Outcome:|Pre-pro<br>(1)|<br>gramtest<br>(4)|<br>Post-pro<br>(3)|gramtest<br>(4)|<br>post-pro<br>(5)|<br>gramtest<br>(6)|
|Treatment|0.021|0.028|0.156|0.153|0.077|0.078|
||(0.091)|(0.097)|(0.106)|(0.110)|(0.051)|(0.051)|
|Girl||-0.033||-0.033||0.009|
|||(0.039)||(0.034)||(0.018)|
|Treatment < Girl||-0.044||-0.022||-0.005|
|||(0.059)||(0.052)||(0.028)|
|Controls|NO|YES|NO|YES|NO|YES|
|Observations|15,984|15,984|15,984|15,984|14.314|14,314|
|R-squared|0.000|0.204|0.006|0.242|0.006|0.031|
|Meanofdep<sup>var in control</sup>|0.000|0.000|0.000|0.000|0.536|0.536|
|<br>P-value:<br>T+TxGirl=0||0.860||0.226||0.190|

Notes: Robust standard errors in parentheses, clustered by village. *** p=<0.01, ** p=0.05, * p<0.1. All analyses are restricted to test-takers, defined as students who were present for pre- and post-program tests in a given year (2012 or 2013). Observations are at the student-subject-level. Controls include 2011 (pre-program) enrollment by gender, cohort fixed effects, and subject (Math, English, and Hind:) fixed effects where indicated, m Columns 2, 4, and 6. Columns 5-6 restricted to students who scored less than the maximum possible score (A) on the pre-program test.
Columns 3-6 show that the program had a large and highly significant effect on test scores at the post-program test in 2012: those in treatment schools performed 0.32 standard deviation higher than those in control schools (Column 4) and students were 22.7 percentage points more likely to improve from the pre-program test (Column 6). The results for 2013 post-program tests, in Panel B, are qualitatively similar, albeit somewhat smaller. On average, students in treatment schools performed 0.15 standard deviation higher than those in the control (Column 4), and are 8 percentage points more likely to improve from the pre-program test (Column 6). There are no significant difference in the program’s effects by gender in either 2012 or 2013 (Columns 4-6).<sup>12</sup> Panel B, Columns 1 and 2 present results on the 2013 pre-program test, where we see no significant differences between treatment and control, suggesting that the learning gains from 2012 did not persist to the next academic school year. We return to this result in our discussion below.

Table 4 pools the analysis across years and all four test administrations—the pre- and postprogram tests in each of 2012 and 2013. Tests of coefficient equality reveal that the treatment effect for the 2012 post-program test is significantly higher than either the preceding test (2012 preprogram test) or immediately subsequent test (2013 pre-program test). While the treatment effect on the 2012 post-program test is larger than the 2013 post-program test, the difference is not statistically significant for all students pooled (p-value=0.226, Column 1) or for boys (pvalue=0.270, Column 2) or girls (p-value=0.207, Column 1) separately.<sup>13</sup>

> 12 Results are qualitatively similar if we aggregate learning results by school-grade, as shown in Appendix Table A3, in which the dependent variable is the average within school-grade among the relevant population.

> 13 We have also performed the analysis pooling across years. We find strongly positive estimated treatment effects on post-program tests for boys, girls, and both genders pooled.

21
Table 4 -- Treatment Effects on Normalized Test Scores across Test Administrations

||AllStudents|Boys|Girls|
|---|---|---|---|
||<br>(1)|(2)|(3)|
|Treatment x 2012 Pre-program test|-0.000|0.008|-0.008|
||(0.088)|(0.094)|(0.090)|
|Treatment x 2012 Post-program test|0.288***|0.295***|0.281***|
||(0.084)|(0.088)|(0.088)|
|Treatment x 2013 Pre-program test|0.006|0.029|-0.018|
||(0.087)|(0.096)|(0.087)|
|Treatment x 2013 Post-program test|0.133|0.144|0.122|
||(0.100)|(0.105)|(0.099)|
|Observations|66,888|33,404|33,484|
|R-squared|0.297|0.298|0.299|
|Mean ofdep var in control|-0.001|0.028|-0.030|
|P-value: T x 2012 Pre-program test= T x 2012 Post-program test|0.000|0.000|0.000|
|P-value: T x 2012 Post-program test= T x 2013 Pre-program test|0.013|0.035|0.008|
|P-value: T x 2012 Post-program test = T x 2013 Post-program test|0.226|0.270|0.207|
|P-value:Tx2013Pre-programtest=Tx2013Post-programtest|0.073|0.102|0.075|

‘Notes: Robust standard errors in parentheses, clustered by village. *** p<0.01, ** p<0.05, * p<0.1. All analyses are restricted to testtakers, defined as students who were present for pre- and post-program tests in the year the test was administered (2012 or 2013). Observations are at the student-subject-level. All specifications include controls for 2011 (pre-program) enrollment by gender, cohort fixed effects, subject (Math, English, and Hindi) fixed effects, and test administration (2012 pre-program test, 2012 post-program test, 2013 pre-program test, and 2013 post-program test) fixed effects.
Table 5 -—- Treatment Effects on Normalized Test Scores by Subject

|PanelA:2012.|AllStudents|Boys|Girls|
|---|---|---|---|
||<br>(1)|(2)|(3)|
|Treatment * Hindi|0.317***|0.323***|0.306***|
||(0.087)|(0.092)|(0.092)|
|Treatment « English|0.256**|0.244*|0.261**|
||(0.121)|(0.124)|(0.127)|
|Treatment x Math|0.369%**<br>(0.095)|0.394***<br>(0.098)|0.340***<br>(0.096)|
|Observations|17,460|8,509|8.951|
|R-squared|0.250|0.268|0.240|
|Mean ofdepvar in control|0.000|0.040|-0.039|
|P-value: T x Hindi=T x English|0.446|0.314|0.618|
|P-value:T x Hindi=T = Math|0.331|0.232|0.562|
|P-value: T = English =T<br>x Math|0.169|0.092|0.351|
|Panel B: 2013|All Students|Boys|Girls|
||(1)|2)|(3)|
|Treatment x Hindi|0.127|0.139|0.119|
||(0.099)|(0.106)|(0.096)|
|Treatment « English|0.159|0.172|0.151|
||(0.125)|(0.127)|(0.129)|
|Treatment x Math|0.136|0.159|0.118|
||(0.109)|(0.114)|(0.109)|
|Observations|15,984|8,193|7,791|
|R-squared|0.247|0.248|0.249|
|Mean ofdepvar in control|0.000|0.004|-0.004|
|P-value:TxHindi=T=English|0.652|0.646|0.679|
|<br>P-value:T x Hindi=T x Math|0.814|0.677|0.980|
|P-value:TxEnglish=T<br>xMath|0.746|0.844|0.676|

Notes: Robust standard errors in parentheses, clustered by village. *** p=<0.01, ** p=<0_05, * p=0_1. All analyses are restricted to test-takers, defined as students who were present for pre- and postprogram tests in the year the test was administered (2012 or 2013). Observations are at the studentsubject-level. All specifications include controls for 2011 (pre-program) enrollment by gender, cohort fixed effects, and subject (Math, English, and Hindi) fixed effects.
Table 6 -- Treatment Effects on Normalized Test Score: Heterogeneity by Pre-Program Test Score

|Panel A: 2012||Normalized Score||
|---|---|---|---|
||All Students<br>(1)|Boys<br>(2)|Girls<br>(3)|
|Treatment = Pre-program testD/E|0.242"|O.279%**|0.205**|
||(0.099)|(0.104)|(0.099)|
|Treatment ~ Pre-program testC|0.352***|0.334***|0.364***|
||(0.071)|(0.075)|(0.072)|
|Treatment ~ Pre-program testB|0.466***|0.462***|0.466***|
||(0.084)|(0.083)|(0.092)|
|Treatment = Pre-program testA|0.056|0.065|0.031|
||(0.046)|(0.044)|(0.064)|
|Observations|17,460|8,509|8.951|
|R-squared|0.614|0.628|0.601|
|Meanofdepvarincontrol|0.000|0.040|-0.039|
|<br>P-value: Effect forD/E = EffectforC|0.154|0.534|0.043|
|P-value: Effect forC=Effect forB|0.082|0.067|0.153|
|P-value: Effect forB =Effect forA|0.000|0.000|0.000|
|Panel B: 2013||Normalized Score||
||All Students|Boys|Girls|
||(1)|(2)|(3)|
|Treatment » Pre-program testD/E|0.152|0.146|0.164|
||(0.133)|(0.136)|(0.139)|
|Treatment = Pre-program testC|0.205**|0.225***|0.189**|
||(0.085)|(0.085)|(0.093)|
|Treatment = Pre-program testB|0.081|0.071|0.091|
||(0.060)|(0.065)|(0.062)|
|Treatment * Pre-program testA|0.004|0.009|-0.005|
||(0.022)|(0.023)|(0.033)|
|Observations|15,984|8.193|7.791|
|R-squared|0.549|0.531|0.348|
|Mean ofdep var in control|0.000|0.004|-0.004|
|P-value: Effect forD/E = EffectforC|0.571|0.429|0.808|
|P-value: Effect forC =Effect forB|0.035|0.011|0.159|
|P-value:EffectforB=EffectforA|0.199|0342|0.128|

‘Notes: Robust standard errors in parentheses, clustered by village. *** p<0.01, ** p<0.05,* p<0.1.Allanalysesare restricted to test-takers, defined as students who were present for pre- and post-program tests in a given year (2012 or 2013). Observations are at the student-subject-level. All specifications include controls for 2011 (pre-program) enrollment by gender, indicators of baseline pre-program test score, cohort fixed effects, and subject (Hindi, English, Math) fixed effects.
Table 7 -- Treatment Effects on School Management Committee Outcomes

|Treatment<br>()|Control<br>(2)|Difference<br>(Treatment -<br>Control)<br>(3)|P-value oftest<br>ofEquality<br>(4)|
|---|---|---|---|
|Number of Meetings<br>4.898|4.239|0.659|0.019|
|(0.192)|(0.202)|(0.278)||
|NumberofImprovement Plans Prepared<br>8.263|6.752|1.510|0.086|
|(0.591)|(0.647)|(0.875)||
|NumberofImprovement Plans Completed<br>5.924|4.752|1.172|0.100|
|(0.476)|(0.527)|(0.708)||
|Observations<br>118<br>|112<br>|||
|‘Notes:Robuststandarderrorsinparentheses.***p<0.01,**p<0.05,*|pk0.1. Observat|ionsareschools.|SS|
# **5. Discussion and Further Results**

# **5.1 Enrollment vs. Retention**

We found relatively large effects of the program on enrollment – especially for girls – in 2012 and 2013. How much of the program effect on total enrollment was driven by new students enrolling versus students being retained?

To answer this question, we match student enrollment records in 2011, to 2012 and 2013 and identify three samples of students: _Baseline_ , _Newly Enrolled_ (students not enrolled in grades three or four in 2011 or 2012, but enrolled in grades four or five in 2012 or 2013, respectively), and _New Cohort_ (students enrolled in grade three in either 2012 or 2013). To match students across 2011, 2012 and 2013 enrollment records, we use information regarding each student’s school, village, gender, age, last and first name, and father’s name. In some cases – for example, when a student name is common or when we are missing demographic information – there are multiple possible matches. Any student who matches to subsequent enrollment records (uniquely or otherwise), is coded as retained to the relevant year.<sup>17</sup> Appendix Table A7 estimates Equation (1) on enrollment numbers, disaggregating total enrollment by sample of student. In 2012, we find that a portion of the treatment effects on enrollment are driven by baseline sample girls – a coefficient of 0.28 (Panel A, Column 3), but the majority of the effects are coming from newly enrolled fourth and fifth grade girls – a coefficient of 1.1 (Panel A, Column 6). We detect no enrollment effects on new third graders (Panel A, Columns 7-9). In 2013, there are positive coefficients on the treatment effect on enrollment for

> 17 Matching baseline students to 2012 yields 97.1 percent unique matches (99.1 percent in the treatment and 95.1 in the control). Similarly, there are 97.6 percent unique matches from the baseline to 2013 (99.3% in the treatment and 95.7% in the control). The difference in the match rate is statistically significant in both years, p=0.001 in 2012, p=0.002 in 2013. The difference in the match rates across treatment and control schools is due entirely to enrollment rosters missing students’ father’s name: 40 schools have 100 percent missing father’s names, of these 37 are control schools.

22
baseline girls (0.194) and for newly enrolled fourth and fifth grade girls (0.626), with the largest effects on new third grade girls (1.036).

As further evidence of retention, we use individual-level data and restrict the sample to baseline students who were enrolled in grades three and four in 2011. We estimate: (7) Enrolledisj � β0 � β1Tj � β2Girlisj � β3Tj � Girlisj � γ<sup>′</sup> Xisj � εisj for student _i_ in school _s_ in village _j_ . We include cohort fixed effects and controls for number of boys and girls enrolled in each school prior to program implementation. Appendix Table A6 presents these results.

Within relevant cohorts, among students in the Baseline Sample in control schools, 77.0 percent were enrolled in 2012, and 58.7 percent were enrolled in 2013. Baseline Sample students in treatment schools were 4.6 percentage points more likely to be enrolled in 2012, a 6 percent increase in retention (Column 1). This effect is statistically significant and concentrated among girls in treatment schools, who were 6.2 percentage points more likely to still be enrolled in school in 2012 (p-value = 0.010) than girls in control schools (Column 2). The effects of retention are similarly high in 2013. The main treatment effect is 6.7 percentage points, or an 11.4 percent increase in the likelihood of being retained over those in the control (Column 3). Again, this effect is higher for girls in treatment schools, who were 8.4 percentage points more likely to re-enroll in school than girls in control schools (p-value = 0.044).

Taken together, these results indicate that the NGO program resulted in higher rates of both retention and new enrollment over the first two years of implementation for girls. Note however, the differences in retention effects are not statistically different across genders.

23
# **5.2 Potential Threat to Validity – Differential Enrollment and Test-Taking**

We found large increases in enrollment and retention among girls due to the program (Table 2, Appendix Tables A7 and A8), and found some positive – and in 2013, statistically significant – program effects on test-taking (Appendix Table A1). While we find no evidence of differences in test-taker characteristics by treatment group in either 2012 or 2013 (Appendix Table 2), we may still worry about differences in unobservables that might bias our learning results.

Appendix Table A9 presents the treatment effects on test-taking (defined as taking both pre- and post-program tests in a given year) by sample type: Baseline, Newly Enrolled, or New Cohort. We find that the main increases in test-taking due to the program are among Newly Enrolled girls in 2012 (Column 6), and Newly Enrolled and New Cohort girls in 2013.

Examining the difference in pre-program learning outcomes between the treatment and control provides some additional insight in terms of selection. Appendix Table A10 presents the estimates from equation (3) on pre-program normalized test score, conditional on taking both preand post-program test scores and disaggregated by sample. We find no evidence that students were negatively (or positively) selected in 2012 (Panel A). There is some suggestive evidence, however, of negative selection among Newly Enrolled girls in treatment schools, who scored somewhat worse on pre-program tests (-0.256 standard deviations lower, p=0.136, Panel B, Column 2). This might suggest that our estimates of the treatment effects on learning would be a lower bound estimate for the true impacts, had the program not had a differential effect on enrollment and testtaking. For completeness, we present learning outcomes in 2012 and 2013 separately by sample in Appendix Table A11.

24
# **5.3 Fade-Out of Program Effects**

The effect of the program immediately after the first year of implementation did not fully persist into the second year of implementation. Among students who were enrolled and test-takers in both 2012 and 2013, the treatment effect on post-program tests in 2012 was 0.278, while the treatment effect on pre-program tests in 2013 was 0.084 (Appendix Table A4, Column 2). The difference between these effects is statistically significant (p=0.090).

Studies on long-run effects have shown that the positive effect on test scores of even successful interventions often fades over time (Banerjee et al. 2007; Andrabi, Das, and Khwaja 2008; Buhl-Wiggers et al. 2019). This may be due to tendencies for programs to “teach to the test”, reinforcing the specific skills that are required for successful test-taking; in this case, education interventions will be more likely to have long-lasting effects when they target core skills (Duflo, Dupas, and Kremer 2011). On the other hand, there may be less enthusiasm for the program over time, or newly learned skills – either by teachers or students – may naturally revert back to preprogram performance without refresher courses or retraining. Longer term gains from an intervention may also re-appear later in life (Chetty et al. 2014).

In our program, Baseline Sample students were exposed to two consecutive years of the program. Our findings—that the program’s positive impacts are smaller and not statistically significant during the second year, despite the continuation of the intervention—contrasts with the other studies that find that learning gains can be cumulative with multiple years of treatment (BuhlWiggers et al., 2019).

Field observations from the research staff’s random visits to treatment schools suggest that two main factors were at play in in the second program year. First, political economy factors changed in the second year, altering volunteers’ incentives in the classroom and reducing their overall 25
engagement. Volunteers were not paid but offered the possibility of future employment as NGO staff. Thus, they may have put in more effort during the first year to keep their options open, until a decision was made regarding whether or not they would receive a position with the NGO. In the second year, after the staffing decision was made, volunteers’ motivation may have dropped both among returning volunteers not granted access to a staff position, as well as among volunteers newly appointed as staff, once their contract was secured. This explanation is in line with the findings of Bold et al. (2016), who show strong positive impacts of a contract teacher intervention in Kenya when implemented by an NGO (0.18 standard deviations), contrasted with null impact when contract teachers were hired by the Ministry of Education.

Second, the somewhat larger classes – as a result of the program’s enrollment drive – may have reduced the effectiveness of the program in the second year (Krueger and Whitmore, 2001). Third, we find significantly lower program gains in the pre-program tests of the second year – at the start of the academic year. There is an established literature in the United States of the decline in academic performance over summer vacation – especially among lower income students (Atteberry and McEachin, 2006; Cooper et al., 1996), yet there is less known in developing countries. Our results suggest that the learning losses over school vacation may be an issue in the context of rural India as well.

# **6. Conclusion**

This paper presents the results of a multi-faceted education intervention conducted in rural Rajasthan, India. The program had three primary aims: increasing participation, learning outcomes, and gender equity at school among a particularly vulnerable population.

26
Many randomized experiments have looked into the effectiveness of programs aimed at either increasing school participation (Bobonis, Miguel, and Puri-Sharma 2006; Duflo, Hanna, and Ryan 2012;) or improving teaching quality (Banerjee et al. 2007; Borkum, He, and Linden 2012; Das et al. 2013; Muralidharan and Sundararaman 2010). Though most learning-targeted interventions, based on an improvement of teaching methods rather than inputs, have shown significant impacts on learning (Banerjee et al. 2007; He, Linden, and MacLeod 2008; Linden 2008), enrollment-targeted interventions have shown zero or small learning effects (Miguel and Kremer 2004; Petrosino et al. 2012). For instance, conditional cash transfers have been shown to improve school enrollment and educational attainment (Galiani and McEwan 2013) but to have limited effects on learning (Behrman, Parker, and Todd 2009).

Our findings – of increased enrollment and learning – suggest that multi-faceted interventions may be able to overcome the possible trade-offs between these two objectives. While a quantity-quality trade-off for schooling would arise if the increase in pupil-teacher ratios hampered a teacher’s ability to improve the learning of all students (Duraisamy et al. 1998), there is a surprising scarcity of rigorous empirical evidence of these types of trade-offs or complementarities in education. Challenging the trade-off assumption, a small number of studies, including Banerjee et al. (2007), have shown an absence of correlation between class size and test scores. Our findings concur with the absence of such trade-off between access and learning. One plausible explanation is that the innovative curriculum, which lies at the core of the program we evaluate here, may be particularly effective at targeting the pedagogical needs of students and counterbalancing the possible harmful effects of enrollment on class size, in line with Banerjee et al. (2007) and Muralidharan, Singh, and Ganimian (2017). The effects of the program on school 27
management outcomes may also be important for its success – evidence has shown that successful school-based management improves both participation and learning (JPAL Policy Bulletin 2017). Although our study is one of the first to evaluate interventions combining enrollment and learning targets at the same time, it is important to emphasize that we do not compare an intervention that is multi-faceted with one that is not. However, this paper suggests that the dual objective of improving both access and learning in primary education may well be achieved through educational policies combining enrollment-targeted interventions with interventions aimed at tailoring curricula to the individual needs of students by teaching at the right level. Rigorous evidence on this statement could emanate from further research investigating the relative effectiveness of interventions targeting (i) enrollment only, (ii) enrollment and learning through additional inputs, and (iii) enrollment and learning through need-tailored curricula.

28