metadata: authors: 0: author: Agostinelli,Francesco
1: author: Avitabile,Ciro
2: author: Bobba,Matteo
count: Mexico
docna: 0: docna: Enhancing Human Capital in Children : A Case
            Study on Scaling
docty: Policy Research Working Paper
subtopic: Educational Sciences,Lifelong Learning,Education For All,Educational Populations,Education for Development (superceded),Inequality
teratopic: Poverty Reduction,Education
abstracts: cdata!: This paper provides new insights on the
            science of scaling. The authors study an educational
            mentoring program with a home visit component implemented at
            scale in Mexico, under different modalities (original and
            enhanced training for mentors) and different situations
            (field experiment and policy implementation). While the
            program was ineffective when implemented by the government
            in its original modality, the enhanced modality boosts
            children&#x2019;s outcomes, both in the field experiment
            and during the government implementation. Higher-quality
            home visits encourage parent/child and parent/community
            interactions, which in turn are found to promote the
            scalability of the program. The work provides new knowledge
            on the socially determined nature of scaling educational programs.
geo_reg_key: 119228,517191
repnme: repnme: Enhancing Human Capital in Children : A
            Case Study on Scaling
display_title: Enhancing Human Capital in Children
            : A Case Study on Scaling
listing_relative_url: /research/2021/05/33046438/enhancing-human-capital-children-case-study-scaling
topicv3: Education, Skills Development and Labor
            Market m1326135 1484,Remedial Education m1326568 1579,Poverty m1326687 695,Education m1327048 470,Coronavirus (COVID-19) m1939478 4828
publishtoextweb_dt: 2021-05-03T00:00:00Z
dois: 10.1596/1813-9450-9647
historic_topic: Poverty Reduction,Education
origu: Education LCR (HLCED)
no_of_pages: 67
author_keys: 0: author_key: NA
1: author_key: 1251237
2: author_key: NA
model_extractions: Policy Research Working Paper 9647 Enhancing Human Capital in Children A Case Study on Scaling _Francesco Agostinelli Ciro Avitabile Matteo Bobba_ Education Global Practice May 2021
Policy Research Working Paper 9647

# **Abstract**

This paper provides new insights on the science of scaling. The authors study an educational mentoring program with a home visit component implemented at scale in Mexico, under different modalities (original and enhanced training for mentors) and different situations (field experiment and policy implementation). While the program was ineffective when implemented by the government in its original modality, the enhanced modality boosts children’s outcomes, both in the field experiment and during the government implementation. Higher-quality home visits encourage parent/child and parent/community interactions, which in turn are found to promote the scalability of the program. The work provides new knowledge on the socially determined nature of scaling educational programs.

This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at cavitabile@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Enhancing Human Capital in Children: A Case Study on Scaling<sup>∗</sup>

Francesco Agostinelli<sup>†</sup> Ciro Avitabile<sup>‡</sup> Matteo Bobba<sup>§</sup> Originally published in the Policy Research Working Paper Series on _May 2021._ This version is updated on _August 2022._ To obtain the originally published version, please email prwp@worldbank.org.

**Keywords** : Children’s Skills; Parental Investment and Community Engagement; Science of Scaling.

**JEL codes** : C90, C93, D02, I3, J1.

> ∗We are grateful to the _Consejo Nacional de Fomento Educativo_ (CONAFE) for the generous collaboration throughout this project, Alonso Sanchez for his initial input into the project, and Miguel Angel Monroy for excellent research assistance. We thank the anonymous referees, as well as Jere Behrman, Horacio Larreguy, John List, Giuseppe Sorrenti, and Stephane Straub for helpful comments and discussions. Avitabile acknowledges financial support for data collection from the Strategic Impact Evaluation Fund (SIEF) of the World Bank and the _Consejo Nacional de Evaluaci ́on de la Politica de Desarollo Social_ (CONEVAL). Bobba acknowledges financial support from the AFD, the H2020-MSCA-RISE project GEMCLIME-2020 GA No 681228, and the ANR under grant ANR-17-EURE-0010 (Investissements d’Avenir program). This study is registered in the AEA RCT Registry and the unique identifying number is AEARCTR-0001645.

> †Economics Department, School of Arts and Sciences, University of Pennsylvania (US). E-mail: `fagostin@sas.upenn.edu`

> ‡World Bank (US). E-mail: `cavitabile@worldbank.org`

> §Toulouse School of Economics, University of Toulouse Capitole (France).

E-mail: `matteo.bobba@tse-fr.eu`
# **1 Introduction**

One of the main challenges in using scientific insights to inform policy decisions comes from the fact that small differences in the implementation of any given intervention can translate into substantial differences in outcomes. Even when programs display large and significant effect sizes in randomized control trials, their success in different situations is far from guaranteed (List, 2022).

This paper contributes to the recent debate about the challenges to scale-up interventions aimed at enhancing human capital in children. In particular, we provide a case study of a mentoring program implemented at scale in Chiapas, the poorest state in Mexico. The program assigns recent university graduates to remote and disadvantaged communities. Among other things, mentors encourage parental involvement in children’s education through home visits. We evaluate the relative effectiveness of two program modalities that differ both in terms of content and the intensity of the training provided to the frontline mentors. The _Original_ modality features a training module focused on curricular knowledge and pedagogical notions, which was initially implemented by the government. The _Plus_ modality embeds a significant change in the training module, which was designed and tailored by our research team to guarantee its operational continuity in the event of a national rollout of the program. The new training protocol includes periodic peer-to-peer meetings during which mentors share their experiences regarding the home visits and their interactions with families.

The evidence on the relative effectiveness of the two program modalities is based on two independent field experiments. The first experiment was directly carried out by the government during the ongoing national rollout of the _Original_ modality of the mentoring intervention. Assignment to the program was randomized across 80 program-eligible primary schools, with 40 getting access to mentors. The results show that the program had no discernible effect on children’s achievement outcomes, as measured by standardized test scores. In the second experiment we randomly assigned both the _Original_ and the _Plus_ modality as well as a control group with no mentoring program across 230 primary schools. After two years of exposure to the mentoring program, the _Original_ program modality displays relatively small and noisy effects on cognitive and socio-emotional scores, as well as on educational achievements when compared to the control group with no mentors. The _Plus_ modality delivers sizable and significant gains in children’s reading scores (+0.32 standard deviations), math scores (+0.24 standard deviations), and socio-emotional scores (+0.20 standard deviations) 1
as well as a large, albeit marginally significant, effect on the probability of enrolling in seventh grade (+12.7 percentage points, out of a basis of 62 percent enrollment in the control group).

The large difference in effect sizes between the two training modalities is corroborated by direct evidence on parental behavior. While both experiments unequivocally display inconclusive evidence on parents’ investment under the _Original_ modality, the _Plus_ modality significantly increases parental engagement both toward the child’s education activities and toward the school community—including volunteering activities, as well as in-kind donations. We further show some evidence that mentors with enhanced training engage in higher-quality interactions with parents during the periodic encounters. In particular, mentors with enhanced training are more likely to inform parents about their children’s learning difficulties, to provide concrete advice to parents on how to tackle these difficulties, as well as to promote parenting styles that are centered around communicating with the child and learning activities. We complement these empirical patterns with qualitative evidence that confirms the role of the peer-to-peer sessions as the driving factor behind both the enhanced parent/mentor interactions and the increased parental engagement.

After the release of this evidence, the government autonomously decided to replace the _Original_ modality of the program with the _Plus_ modality for all its primary schools throughout the country, including those that were part of the experimental evaluation. This reform provides us with a unique opportunity to study the determinants and mechanisms of scaling. One of the key situational differences between the experimental setting and the policy implementation comes from the fact that several schools in this context are at risk of closure, an event that has disruptive consequences for children’s learning and their educational trajectories.<sup>1</sup> While the intense monitoring during the experimental evaluation has minimized the extent of school closures, this high-stakes implementation feature may compromise the success of the program under the business-as-usual conditions. However, two years after the rollout, none of the schools that received a mentor during the government implementation closed, while approximately 10 percent of the other schools did.

We next zoom into the relationship between exposure to the mentors and school closures in order to study the sources of scalability of the program. Parents play an important role in the community-based schooling system under study (Gertler et al., 2012). We sketch a

> 1The importance of keeping schools open for the development of children has recently gained momentum in educational studies on the impact of the COVID-19 lockdowns on schooling outcomes (see, e.g., Agostinelli et al., 2022; Engzell et al., 2020; Maldonado and De Witte, 2020).

2
simple model of parental investment with local spillovers to formalize the idea that parents have an active role in promoting educational opportunities, and that educational investment at the community level are a socially determined outcome (List et al., 2020). We show that the extent to which an educational program preserves or loses impact at scale depends upon its ability to promote parental coordination and engagement in the local community. We empirically corroborate these predictions by leveraging the changes in community-level parental engagement induced by the experiment. Using this variation, we document that parents prevent schools from closing, and as a consequence promote the effectiveness of the mentoring intervention during the government implementation. We find that an increase of half standard deviations in parental engagement decreases the probability of school closures by 11 percentage points over the subsequent two years. Our qualitative data from in-depth surveys of mentors and local instructors further corroborate the role of parents in guaranteeing the continuation of educational activities in the communities, in a context with poor school infrastructure and where schooling activities are often disrupted.

Finally, we study the educational impacts of the policy reform across the overall population of schools in the state of Chiapas. The assignment of the mentoring program under the _Plus_ modality at scale was done through a rotating scheme with a priority-based mechanism. We exploit the quasi-experimental variation in the program rollout once we condition on the set of eligibility criteria officially used by the government. After providing evidence that this variation appears conditionally “as-good-as random” via various placebo tests, we show that the program was successful in the schools that were previously part of the evaluation sample as well as in the rest of the schools in Chiapas. Within the evaluation schools, the marginal effect of the _Plus_ modality after one year of government implementation on the probability of enrolling in seventh grade is +5.4 percentage points. The cumulative effect of continuous exposure to the program for three years (two years under the experiment and one year under the government) implies that the enrollment rates in these disadvantaged and rural areas achieve the secondary school enrollment rates in urban Mexico (95 percent). For the much larger sample of schools that did not participate in the experimental evaluation, the results show a positive effect on secondary school enrollment, with an average program impact of 4.5 percentage points under the _Plus_ modality at scale. We further document positive effects of the program on child literacy, which imply a reduction of illiteracy rates by 20 percent with respect to the sample mean, as well as a decrease in school closures that is remarkably similar to the corresponding impact of the program under the experimental assignment. Taken together, these findings corroborate the effectiveness of the intervention in increasing 3
schooling opportunities under the new situation created by the policy implementation.

**Relationship to Literature.** There is a consensus in the literature that gaps in family investment and parent/child interactions are behind the gaps in children’s achievements among different socio-economic groups (Cunha et al., 2010; Fryer et al., 2015; Agostinelli and Wiswall, 2016). Moreover, the literature provides ample evidence that successful home visit and mentoring programs, although implemented in very different contexts, share the common outcome of stimulating parental investment and parent/child interactions. Several studies in developing countries document increased cognitive and socio-emotional outcomes for children in the first years of life (see Attanasio et al., 2022b, for a complete review). In the United States, successful interventions like the Perry Preschool project, and the Carolina Abecedarian project show improvements in the home environment (see Heckman and Mosso, 2014, for a complete review). The quality of child/home-visitor interactions and parent/home-visitor interactions are found to be key ingredients for boosting the impact of home visiting programs (Carneiro et al., 2019; Heckman and Zhou, 2021; Zhou et al., 2021). However, little is known on the role of family and community interactions in sustaining program impacts at scale.<sup>2</sup> Our study fills this gap by providing direct evidence on how enhanced training for home visitors promotes higher parental engagement and interactions at the community level, which in turn promotes the success of the program at scale. To the best of our knowledge, we are the first to highlight that parents can act as means of scalability, which has implications for the design and the evaluation of scalable educational interventions that actively include parents in the learning process.

In recent years, scholars and policy makers alike have been increasingly concerned about the ability of field experiments to inform policy decisions, given that experimental interventions that have been found effective often fail to live up to their promises when implemented at scale by governments or firms (Bold et al., 2018; Cameron et al., 2019; Muralidharan and Singh, 2020). Based on the insights of recent work (Banerjee et al., 2017; Muralidharan and Niehaus, 2017; Davis et al., 2017; Mobarak and Davis, 2021), our educational intervention is well-equipped to overcome some of the major threats to scalability. We highlight the infor-

> 2For example, Zhou et al. (2021, p. 90) state: “ _The body of research discussed above clearly identifies the key mechanism by which home visiting programs positively impact short-term and long-term outcomes for children: fostering engagement between caregiver and home visitor to improve the caregiver’s quality and frequency of caregiver–child interaction, thereby fostering child development. This volume, including this chapter, seeks to move the field toward understanding how to effectively scale up promising interventions and inspire more research on the subject._ ” In their recent review, Attanasio et al. (2022b, p. 886) raise another important issue: “ _[S]calability does not only refer to the financial cost of running these interventions but also to the ownership and acceptability of the intervention by the community that is targeted. How should interventions be designed and delivered to take account of this important distinction?”_

4
mative features for scalability of our experimental design following the key points analyzed in Al-Ubaydli et al. (2020). First, we harness the value of replication by drawing joint inference from two independently run field experiments on different and representative samples of schools that share one of the two program modalities. Second, the field experiments were run while the program was already at scale and in close collaboration with the government agency that was later in charge of the program’s rollout and policy implementation. Third, the government agency and the research team designed the _Plus_ modality together, bearing in mind the financial and human resources constraints of the context under study. Finally, the relatively large units of randomization (schools/communities) take into account possible local spillover effects that often arise in the context of interventions evaluated at scale (Miguel and Kremer, 2004; Bobba and Gignoux, 2019; List et al., 2020).

# **2 Context and Experimental Design**

The _Consejo Nacional de Fomento Educativo_ (CONAFE) is a semi-autonomous government agency responsible for providing schooling services in highly marginalized communities of Mexico with a population below 2,500 inhabitants. In those communities, CONAFE offers all education services from pre-school until the end of lower secondary school (hereafter, we refer to the population of CONAFE schools as schools). In 2013, these schools accounted for 10 percent of the roughly 99,000 primary schools and 7 percent of the 38,000 lower secondary schools across the 31 Mexican states. About 20 percent of the schools are located in Chiapas, the Mexican state with the highest incidence of poverty in the country (CONEVAL, 2018).

Primary schools typically have a single multi-grade classroom with 10–15 students. Instructors are generally community residents between 15 and 29 years old. Only 2.6 percent report having a college degree, while 19 percent report having only completed lower secondary education. Instructors are supposed to receive between five and seven weeks of training, but more than half report four weeks of training or less. They receive a stipend of MXN $1,427 per month (US $95 in 2015). After one year of service in the community school, instructors become eligible to receive a scholarship of MXN $982 per month for up to 30 months, which is conditional on enrolling in a higher education institution. As a result of the very low compensation and extremely challenging conditions, about one quarter of the instructors drop out before completing the first school year (Bando and Uribe, 2016).

5
# **2.1 The Mentoring Program**

In 2009, the government launched the “Mobile Mentors” ( _Asesores Pedagogicos Itinerantes,_ API henceforth) program as an attempt to improve the quality of education provision in primary schools located in the most-deprived communities. Initially, the program was implemented in 11 states, but starting in 2012, it was extended to all 31 states in Mexico. The mentors are selected from recent university graduates (the program was advertised both during on-campus visits and announcements through the media). Preference is given to applicants with degrees in pedagogy, psychology, sociology, and social services who have previous experience as community instructors and who speak an indigenous language. They are usually hired for a two–year period and receive a monthly salary of MXN $5,000 (US $332 in 2015).

After a week-long training session focused on curricular knowledge and basic notions of pedagogy, the mentors are assigned to schools on a rotating-based algorithm, which gives differential priority across the school communities according to four criteria: (i) at least 30 percent of the students are classified as “insufficient” in the National Standardized test; (ii) at least six students are enrolled, (iii) there are high levels of poverty and marginalization in the respective municipalities; and (iv) the school has not received a mentor in previous academic cycles. Mentors meet with their supervisors every two months in two-day sessions throughout the school year. In December 2018, there were 1928 mentors deployed throughout Mexico, and the largest share is in Chiapas (20 percent).

The mentors periodically conduct home visits to provide parents with information on their children’s progress in school and promote their participation in school activities. In addition to working on behavioral issues directly with the children, the mentors are supposed to address them with parents as part of the home visits. Each mentor is assigned to a maximum of six students for individual (one-on-one) remedial education sessions that take place after the regular instructional time. Student eligibility for the remedial sessions is determined by a diagnostic evaluation at the beginning of the school year and an additional exam to assess the grade to which the student’s knowledge corresponds. During regular school hours, the mentor is supposed to observe and take notes on the teaching practices of the community instructor, help her with the students who have learning difficulties, and work outside the classroom with students who are unable to attend the remedial sessions in the afternoon.

6
# **2.2 Two Field Experiments**

In this section we describe the field experiments that we use to evaluate the impact of two different modalities of the API program. A full description of the different data sources used throughout the empirical analysis is provided in Appendix A. Because both experiments were run at scale within the infrastructure of the existing program, the recruitment process and the assignment mechanism of the mentors are the same between experimental and nonexperimental schools.

**First Experiment.** As part of an effort to evaluate a broader set of interventions targeted to families and schools in disadvantaged communities (Mart ́ınez, 2012), in 2010, the government undertook the first impact evaluation of the API program. Eighty primary schools are selected among those that met the eligibility criteria for the program across four Mexican states (72 in Chiapas). Assignment to the API program was randomized at the school level using a block design, with the strata represented by the Mexican states where schools are located. Forty schools were assigned to receive the API program starting from the 2011– 2012 school year while the remaining half of the schools were assigned to the control group without mentors.

A mid-line survey collected after one year of the API assignment recorded parental behaviors and investments for 208 parents in 73 schools (the enumerators were not able to reach the parents in seven schools). Student outcomes were measured two years after treatment assignment through the results in the national standardized test for students in grades three through six. Due to the incomplete take-up of the test—mainly due to the opposition from the teachers’ unions in some states—we were able to match 70 schools with 599 test score records out of the subsample of 73 schools with parental outcomes. Both sources of sample attrition are orthogonal to treatment assignment. Five of the unmatched schools were in the treatment group and five were in the control group. Table B-1 shows the balance for the original sample as well as the nested samples with parental outcomes and student outcomes between the treatment and the control groups across mean community-level and school-level characteristics measured in the year before the start of the first experiment.

**Second Experiment.** In 2014, as part of a World Bank project, we designed and evaluated

7
an alternative training modality aimed at strengthening the original mentoring program.<sup>3</sup> The API _Plus_ modality embeds all the features of the _Original_ modality, with two significant changes in the training module. First, it entails two weeks rather than one week of initial training. The extra week is focused on hands-on strategies to improve students’ reading and math competencies. Second, mentors attend an additional day during each one of the bimonthly meetings throughout the school year. The schedule of the extra day is organized around peer-to-peer sessions in which mentors share experiences and design common strategies to better address the most pressing issues. Among the large array of possible improvements in the design of the program, enhanced training was ex-ante scalable and yet a promising alternative from the perspective of the government. The _Plus_ modality entails a cost per child of US $332 as opposed to US $285 per child for the _Original_ modality. These cost figures are very much in line with another recent government-run intervention that targeted both children and parents through home visits in Colombia (Attanasio et al., 2022a).

We randomly selected 230 schools in rural Chiapas from a set of schools that were not previously part of the API program. Assignment of the program was carried out using a randomized block design at the school level, with the strata represented by the deciles of the 2012 school-average in a national standardized achievement score in the Spanish test (see Appendix A.1). As a result, 60 schools were assigned to the API _Plus_ , 70 schools were assigned to the API _Original_ modality, and the remaining 100 schools were in the control group with no API intervention.

We rely on both administrative and survey data sources as well as qualitative interviews for the second field experiment. Most of these variables are shown in Table B-2, and they are balanced with respect to treatment assignment. The data collection took place by the end of the second school year after the inception of the API program in the evaluation sample. By that time, two schools out of the original 230 schools in the evaluation sample had closed, while the program could not be put in place in another four schools due to high political instability. Within the remaining 224 schools, one quarter of the community instructors reported eight or fewer months of tenure in the school, and only 56 out of the original 126 mentors were working in the same schools to which they had been originally assigned. All

> 3The _Original_ modality in the second experiment is meant to track the benchmark intervention with two minor differences. First, the ability to speak the main indigenous language in the community would become the most important criterion for the assignment of the mentors across program-eligible communities. Second, the supervisors of the mentors would receive a salary increase in exchange for a mandatory increase in the frequency of their visits to the targeted communities.

8
these outcomes are well balanced across treatment arms. Out of the six schools that dropped out of the sample, two schools were in the control group, two were in the _Original_ group, and two in the _Plus_ group. The _p_ -values of the Komolgorov-Smirnoff statistic for the equality of the distributions of work experience in the school of the community instructors in each treatment arm and the control group are 0.773 and 0.892, respectively. The _p_ -value of the _Plus_ - _Original_ difference in the share of mentors who drop out from the program during the experiment is 0.957. There is no evidence of composition changes between the _Original_ and _Plus_ groups induced by mentor turnover (see Table B-3).

The effective sample size of the second experiment is 1,045 children/parents and 224 schools (see Appendix A.2 for further details on the sampling design). We use the Early Grade Reading Assessment (reading score) and the Early Grade Math Assessment (math score) as our main measures of children’s cognitive achievement. Those are individually administered student assessments that have been conducted in more than 40 countries and in a variety of languages (Dubeck and Gove, 2015; Platas et al., 2016). While these instruments are typically applied to students in first, second, or third grade, we administer them to third through six grade students to account for the large learning gaps of the children in our sample. The school-average standardized scores in math and Spanish as measured in the school year prior to the introduction of the second experiment are, respectively, 0.5 and 0.7 standard deviations below the national averages.<sup>4</sup> To measure the impact of the intervention on socio-emotional skills, we consider a collection of thirty-two behavioral issues as reported by a caregiver, which resembles the questionnaire in the Children section of the National Longitudinal Study of Youth (CNLSY-79), such as antisocial behavior, anxiety/depression, headstrongness, hyperactivity and peer conflicts (for details, see Appendix A.2). The resulting behavioral problem index is re-scaled in such a way that higher values are associated with fewer behavioral issues (socio-emotional score). The survey also contains a module on instructors’ characteristics as well as pedagogical practices collected through an adapted version of the Stallings Classroom Snapshot (Bruns and Luque, 2015), a module on parental attitudes and investment toward children’s education, as well as information about the mentors’ activities in the communities, among others. To better interpret our results, we standardize most of the survey-based outcome variables using the mean and the standard

> 4Only 5 percent of the children in our sample score at the maximum of the scale in two or more subdomains of the reading score (out of eight subdomains) and in three or more subdomains of the math score (out of a total of seven subdomains). Unlike the first experiment, we cannot leverage the national standardized test scores for the second experiment since the test ceased to be universal during the period of interest (after 2014).

9
deviation observed in the control group.

In addition, we access separate administrative data on students’ records that we use to construct an indicator for enrollment in seventh grade, which is the first grade in lower secondary school. The sample reduces to 468 sixth graders in 182 schools, who are deciding whether to transit to secondary school. This sample reduction is due to the multi-grade aspect of the schooling system, where student composition among grades in each school is not homogeneous in size. Missing schools in this analysis are balanced among treatment arms. The choice of this cohort of students is meant to maintain the same length of exposure to the API program of the main sample of the analysis.<sup>5</sup> Finally, we conducted a series of in-depth interviews in the spring of 2022 for a small and representative subsample of 16 mentors and 12 community instructors who were part of our study.<sup>6</sup> This qualitative evidence proves useful to complement the quantitative analysis and to shed further light on the mechanisms through which the mentoring intervention affects students and parents as discussed in Section 3.3, as well as on the role of parents as means of scalability as discussed in Section 5.2.

# **3 Experimental Evidence**

In this section, we report OLS estimates from separate regression models for each experiment on the treatment assignment indicators for the _Original_ and _Plus_ modality after two years of exposure to the mentoring program. All models include the strata indicators that account for the block randomization design (see Section 2.2) as well as few individual characteristics such as students’ age and ethnicity. We control for interview week fixed effects, which account for changes in weather and political conditions, as well as indicators for the different teams of enumerators who administered the survey across the communities in our sample. The error terms are clustered at the school level, which represents the unit of randomization of the treatments. We complement the usual asymptotic inference with two alternative procedures. First, we display _p_ -values based on randomization inference, which are accurate

> 5The distribution of missing schools in the analysis of transition to secondary school is 18 schools in the control group, 14 in the _Original_ group and 16 in the _Plus_ group. Due to the different individual identifiers, we are not able to match this dataset to the survey data. The estimates reported in Table B-6 document no program effects on grade repetition and attrition, which suggest that conditioning on grade attainment is not problematic in our context.

> 6Appendix A.3 reports more details about these interviews. Tables B-4 and B-5 show that the characteristics of these survey respondents are broadly comparable to those of the mentors and the local instructors in our main sample.

10
even with a small number of clusters. This may be especially relevant in the context of the first experiment, which had fewer schools per treatment arm than the second experiment. Second, given the large array of hypotheses considered throughout the analysis, we also provide _p_ -values that are adjusted for multiple hypothesis testing across different families of outcomes (List et al., 2019).

# **3.1 Children’s Outcomes**

Table 1 and the first row of Table 2 display the impacts of the _Original_ modality on children’s outcomes, as measured by individual test scores collected two years after the introduction of the mentoring program in each experiment, respectively. For the first experiment, the outcome variables shown in Table 1 are based on administrative records of sixth graders in a national standardized test. For the second experiment, we collect our own measures of cognitive and socio-emotional skills (first to fourth columns of Table 2), as the national standardized test was terminated in 2014.<sup>7</sup> In spite of the differences in measurement of the outcome variable, the separate analyses of the two experiments show consistently inconclusive evidence regarding the effectiveness of the _Original_ modality of the mentoring intervention. Depending on the outcome, the effect of the program in the first experiment ranges from positive to negative and is not statistically different from zero. The effect size of the estimated treatment effect on the overall index for student achievement (column 4 of Table 1)—a Generalized Least Squares (GLS)-weighted average across the three subject tests that increases the power of the analysis (O’Brien, 1984)—is negative, small and imprecise.<sup>8</sup> Effect sizes are consistently positive and slightly more precise in the second experiment, although none of the estimated coefficients gets close to the conventional significance levels. The impact on the GLS-weighted overall index for student achievement across the two cognitive measures and the socio-emotional score is 0.12

> 7Another national standardized test was administered by the National Institute for the Evaluation of Education (INEE) starting in 2015, the PLANEA National Plan for Learning Evaluation. While the national test that we employ in the first experiment (ENLACE) was administered to all Mexican students in grades three through six through the year 2013 (see Appendix A.1), PLANEA scores are collected only on sixth graders in a random sample of students within schools.

> 8The GLS weighting procedure increases efficiency when compared to other summary indices by ensuring that outcomes that are highly correlated with each other receive less weight, while outcomes that are uncorrelated and thus represent new information receive more weight. This procedure is more powerful than other popular tests in the repeated-measures setting. Also, missing outcomes are ignored when creating the GLS-weighted score. Thus this procedure uses all the available data, but it weights outcomes with fewer missing values more heavily.

11
Table 1: Children’s Achievement—First Experiment

||Reading Score|Math Score|Science Score|Overall Index|
|---|---|---|---|---|
|API Original|-0.053|0.083|-0.082|-0.022|
||[0.737]|[0.655]|[0.585]|[0.902]|
||_{_0.750_}_|_{_0.669_}_|_{_0.591_}_|_{_0.910_}_|
||(0.779)|(0.739)|(0.717)|(0.878)|
|Number of clusters|70|70|70|70|
|Observations|599|599|599|599|

_Notes_ : This table shows OLS estimates and the associated _p_ -values on student outcomes measured after two years of exposure to the mentoring program under the first experiment run by the government. For detailed descriptions of the test scores used in this table, see Appendix A.1. The dependent variables are standardized with respect to their means and the standard deviations in the control group. _p_ - values reported in brackets refer to the conventional asymptotic inference. _p_ -values reported in braces are computed using randomization inference (randomization- _t_ ). _p_ -values reported in parentheses are adjusted for testing the null impact of API Original across the five outcomes shown in the tyable through the stepwise procedure described in Romano and Wolf (2005a,b, 2016). All _p_ -values account for clustering at the school level.

standard deviations—a non-negligible effect size that is nonetheless not statistically different from zero. The effect of the _Original_ modality of the mentoring program on the transition rates to lower secondary school are shown in the last two columns of Table 2. Estimated effect sizes are noisy and relatively small in magnitudes, ranging between an increase of seven and eight percentage points out of a basis of 62 percent enrollment rate in seventh grade in the control group.

The second row of Table 2 displays the estimated coefficients for the average impact of the _Plus_ modality of the API program when compared to the control group. Children who are enrolled in a school that received the _Plus_ modality increased their reading scores by 0.32 standard deviations. We can reject the hypothesis of a null effect of API _Plus_ at the 99 percent confidence level across all three different inference procedures. Quantitatively, the API _Plus_ effect is approximately 2.5 times higher than the effect of the API _Original_ . The difference between the two modalities is statistically different from zero at the conventional 95 percent level in two out of three inference procedures.

We find similar patterns when we look at math scores, which show a sizable effect of the _Plus_ modality with an estimated treatment effect of 0.24 standard deviations. This effect is precisely estimated, and we can reject the hypothesis that the two treatment arms have the same effect at the 95 percent confidence level in two out of three cases. The API _Plus_ program also generates a sizable improvement in the socio-emotional score of 0.2 standard deviations.

12
Table 2: Children’s Achievement and Attainment—Second Experiment

|||Surve|y-Based Test Scores||Enroll 7th|Grade (1 = yes)|
|---|---|---|---|---|---|---|
||Reading|Math|Socio-emotional|Overall Index||age _≥_13|
|API Original|0.126|0.056|0.071|0.124|0.073|0.081|
||[0.104]|[0.455]|[0.418]|[0.187]|[0.255]|[0.519]|
||_{_0.138_}_|_{_0.483_}_|_{_0.440_}_|_{_0.218_}_|_{_0.283_}_|_{_0.573_}_|
||(0.147)|(0.554)|(0.554)|(0.234)|(0.312)|(0.469)|
|API Plus|0.315|0.237|0.199|0.366|0.124|0.298|
||[0.001]|[0.008]|[0.022]|[0.001]|[0.074]|[0.030]|
||_{_0.001_}_|_{_0.012_}_|_{_0.030_}_|_{_0.001_}_|_{_0.084_}_|_{_0.053_}_|
||(0.001)|(0.005)|(0.011)|(0.001)|(0.032)|(0.032)|
|API Original = API Plus|[0.043]|[0.043]|[0.178]|[0.020]|[0.469]|[0.134]|
||_{_0.086_}_|_{_0.115_}_|_{_0.225_}_|_{_0.024_}_|_{_0.570_}_|_{_0.229_}_|
||(0.045)|(0.045)|(0.098)|(0.023)|(0.376)|(0.157)|
|Number of clusters|224|224|224|224|182|76|
|Observations|1044|1044|1045|1045|468|106|

_Notes_ : This table shows OLS estimates and the associated _p_ -values on student outcomes measured after two academic years of exposure to the API program under the second experiment designed and implemented by the authors in collaboration with the government. For detailed descriptions of the test scores used in this table, see Appendix A.2. The dependent variables in the first four columns are standardized with respect to their means and the standard deviations in the control group. The dependent variable in the last two columns is computed from administrative school records (see Appendix A.1). _p_ -values reported in brackets refer to the conventional asymptotic inference. _p_ -values reported in braces are computed using randomization inference (randomization- _t_ ). _p_ -values reported in parentheses are adjusted for testing each null hypothesis (null impact of API Original, API Plus, and the comparison) for the two different families of outcomes (survey-based and administrative data) through the stepwise procedure described in Romano and Wolf (2005a,b, 2016). All _p_ -values account for clustering at the school level.

While the difference with respect to the _Original_ modality is not statistically significant, the larger effect of the _Plus_ modality is consistent with qualitative evidence documenting that mentors with enhanced training shared more effective strategies to best deal with children’s emotions during the bimonthly peer-to-peer sessions (see Appendix A.3). The effect size of the _Plus_ modality on the GLS-weighted index of achievement is very large, 0.37 standard deviations—precisely estimated, and statistically different at the 95 percent level from the effect of the _Original_ modality.<sup>9</sup> The last two columns in Table 2 report the estimated effects on the average transition rate to secondary school. Less than two-thirds of the sixth graders in the control group enroll in seventh grade, while the corresponding national average is 95 percent. The _Plus_ modality

> 9In Table B-7 we report the results by sub-domains of the reading scores (panel A), math scores (panel B). While the estimates are erratic and not statistically significant for the _Original_ modality, the _Plus_ modality is shown to increase students’ proficiency in reading across various domains (familiar-word reading, reading comprehension, and dictation). There are no improvements in sound-related questions (initial sound and initial name), which is probably due to the fact that children whose first mother tongue is an indigenous language might struggle to capture Spanish alphabet pronunciation. For math scores, the _Plus_ modality seems particularly effective on numbers’ identification and discrimination as well as additions. There are no improvements in more involved tasks such as problem solving and shape recognition. Similarly, in Table B-8 we report the effects of the two program modalities for each individual component of the socio-emotional score.

13
increases the probability of a child’s enrolling in seventh grade by 13 percentage points. Although marginally significant, the effect is quantitatively sizable, as it represents a 20 percent increase in the share of students who transit to secondary school relative to the mean in the control group. This effect more than doubles in size when we focus on the subsample of over-aged sixth graders (13 years old or more, sixth column), and it persists one year after the second experiment (see Figure B-1). Given the prevalence of child labor in Chiapas, this result for older children is particularly important in terms of life-cycle opportunities. While the estimated coefficient of the _Plus_ modality in the last column of Table 2 is significant at the 95 percent level for two out of three cases, the _p_ -values reported in the third row show that we cannot reject that it is different from the corresponding estimate of the _Original_ modality.

# **3.2 Parental Investment and Behavior**

Home visits are a key component of the mentoring intervention under study. The goal of these home visits and repeated family/home-visitor interactions is to increase parental awareness about their children’s educational trajectories. Table 3 presents the average impact of the program on GLS-weighted indices of parental behavior and investment in their children’s education (see Appendix A.2). Panel A displays the estimates of the _Original_ modality in the first experiment, while Panel B shows the corresponding figures for both the _Original_ and _Plus_ modality in the second experiment. Under the _Original_ program, consistently across experiments, the estimates are not statistically different from zero, with signs of the coefficients that range from positive to negative and effect sizes on the overall index of -0.03 and 0.1 standard deviations. Instead, parents appear to be systematically more invested in their children’s education activities under the _Plus_ modality of the program. The estimates reported in the second row of Panel B document that mentors with enhanced training are more effective in boosting parental engagement, both toward the school and directly with the child. The point estimates are positive throughout; three out of four coefficients are statistically significant at the 95 percent level with a very large effect size for the overall index of parenting practices of 0.36 standard deviations. While we can reject the null hypothesis of equal treatment effects on most parental outcomes shown in Table 3, we refer the reader to Section 4.1 for a more thorough discussion on hypothesis testing under both experiments 14
Table 3: Parental Investment and Behavior

||Engage at School|Manage School R|esources|Engage With Child|Overall Index|
|---|---|---|---|---|---|
|||Panel A:|First Ex|periment||
|API Original|0.198|-0.135||0.149|0.101|
||[0.259]|[0.415]||[0.399]|[0.580]|
||_{_0.261_}_|_{_0.422_}_||_{_0.399_}_|_{_0.578_}_|
||(0.338)|(0.511)||(0.511)|(0.511)|
|Number of clusters|73|73||73|73|
|Observations|208|208||208|208|
|||Panel B:|Second E|xperiment||
|API Original|-0.188|-0.124||0.167|-0.034|
||[0.049]|[0.176]||[0.015]|[0.684]|
||_{_0.058_}_|_{_0.197_}_||_{_0.015_}_|_{_0.630_}_|
||(0.067)|(0.205)||(0.021)|(0.704)|
|API Plus|0.217|0.087||0.353|0.359|
||[0.034]|[0.344]||[0.001]|[0.001]|
||_{_0.037_}_|_{_0.247_}_||_{_0.001_}_|_{_0.001_}_|
||(0.055)|(0.388)||(0.001)|(0.002)|
|API Original = API Plus|[0.001]<br>_{_0.001_}_|[0.056]<br>_{_0.056_}_||[0.029]<br>_{_0.158_}_|[0.001]<br>_{_0.001_}_|
||(0.002)|(0.036)||(0.036)|(0.001)|
|Number of clusters|224|224||224|224|
|Observations|1045|1045||1045|1045|

_Notes_ : This table shows OLS estimates and the associated _p_ -values on survey-based measures of parental behavior measured after two years of exposure to the API program. Panel A refers to the first experiment run by the government. Panel B refers to the second experiment designed and implemented by the authors in collaboration with the government. For detailed descriptions of the individual components of the summary measures of parental engagement used in this table, see Appendix A.2. _p_ -values reported in brackets refer to the conventional asymptotic inference. _p_ -values reported in braces are computed using randomization inference (randomization- _t_ ). _p_ -values reported in parentheses are adjusted for testing each null hypothesis (null impact of API _Original_ , API _Plus_ , and the comparison) for the two different families of outcomes through the stepwise procedure described in Romano and Wolf (2005a,b, 2016). All _p_ -values account for clustering at the school level.

# (see also Table 5).<sup>10</sup>

Overall, the results presented in these two sections show that the API intervention had differential impacts according to the training received by the mentors. While the _Original_ modality does not significantly boost any of the outcomes of interest across two independently run field experiments, the _Plus_ modality is shown to generate sizable average effects on children’s cognitive and socio-emotional scores, on schooling attainment, as well as on parental engagement toward their children’s education. In the next section, we leverage de-

> 10We also estimate the impacts of both the _Original_ and _Plus_ modalities for each of the individual measures of the parental behavior collected in the survey that have been aggregated in the summary measures displayed in Table 3. Table B-9 reports he results, which are broadly comparable to the estimates discussed in the text. They show large and significant effects for the _Plus_ modality on food donations to the instructors, the management of the school resources, help with homework, enrolling their children in extra-curricular activities, expecting their children to complete secondary education or more, and meet periodically with the instructor.

15
tailed survey information to provide direct evidence on the possible channels through which the API _Plus_ program enhance children’s outcomes.

# **3.3** **_Plus_ vs.** **_Original_ : Channels**

We start by evaluating the role of the remedial education sessions. The estimates displayed in Table B-10 suggest that there is no differential effect across the four children’s outcomes ( _p_ - values = 0.766, 0.675, 0.639, and 0.937) in the relative impact of the two training modalities between children who are more or less likely to be eligible for the remedial sessions (see also Figure B-2). Although the design of the second experiment does not allow us to directly isolate the direct effect of the remedial education sessions within each API modality, this evidence suggests that such mediating factor is unlikely to explain the differential impact between the _Plus_ and the _Original_ documented in Table 2.<sup>11</sup> We next consider the role of the pedagogical practices of the community instructors. Table B-11 reports estimates of the effect of the two API modalities using data at the instructorschool level (the sample average number of instructors per school is 1.2 in the school year prior to the start of the second experiment) on four summary measures of pedagogical practices based on GLS-weighted indices across an array of instructor-student interactions (for details, see Appendix A.2). The results show erratic patterns of positive and negative signs with no statistically significant effects of either API modality. The overall index of pedagogical practices reveals a non-negligible negative effect of 0.18 standard deviations for the _Plus_ modality, indicating, if anything, a crowding-out effect of the presence of the mentors on instructors’ job effort. The effects are quantitatively and statistically similar between API modalities across the different pedagogical practices, as shown in the third row of Table B-11.

Finally, we study the role of the mentor/parent interactions during the home visits as a potential mechanism behind the large and positive effect of the _Plus_ modality. Panel A in Table 4 displays the estimated differences across the two API modalities on selected survey variables when 591 parents were asked about the frequency and content of their interactions with the mentors over a period of two months prior to the survey. The number of observations varies across the columns in Panel A due to some of the parents not responding to the survey questions. Missing values for each outcome are balanced with respect to the assignment of

> 11The correlation between the school-level rankings as implied by the average diagnostic test and the math and reading scores is 0.51 and 0.52, respectively. In the absence of randomization across the different components of the intervention within each modality, the direct effect of the remedial education sessions cannot be separately identified from heterogeneous treatment effects by academic achievement.

16
the _Plus_ modality ( _p_ -values = 0.746, 0.183, 0.442, 0.517, 0.539, and 0.575). In spite of quite noisy estimates due to the sample attrition and the reduced sample size—parents in the control group cannot be part of this analysis by design—the evidence does show a systematic pattern. Over a two-month period, mentors in the _Plus_ modality met one time more with parents at school and 0.7 times more at home compared to those in the _Original_ modality (sample means in the _Original_ group are five and three, respectively). The GLS-weighted index shown in the third column documents that the quantity of parent-mentor interactions increased by 0.36 standard deviations under the _Plus_ modality, which is significant at the 10 percent level. Columns 4 and 5 of Panel A show marginally significant estimates on two measures of the quality of the interactions between parents and the mentors: (i) an indicator variable for whether the mentors have informed parents about their children’s learning difficulties, (ii) and whether the mentors provide concrete advice to the parent on how to tackle these difficulties. The effect sizes are large for both outcomes, implying a 14 percent increase in the probability of informing parents relative to the respective sample means in the _Original_ group (70 percent). The estimated coefficient for the GLS-weighted quality index is 0.25 standard deviations, which is significant at the 90–95 percent level depending on the inference procedure.

Panel B in Table 4 shows the effect of the _Plus_ modality on different competencies, or “parenting style,” that the mentors report to have promoted during their encounters with parents. This information was collected during a follow-up interview at the end of the field experiment. Of a total of 126 mentors between the _Original_ and _Plus_ modalities, enumerators were able to interview 107 of them. The attrition of survey participation of mentors is unrelated to the treatment assignment ( _p_ -value = 0.514). For further details on the survey of mentors, please refer to Appendix A.2. Mentors with enhanced training are more inclined to foster attitudes that are centered on educative parenting styles, such as communicating with the child (first column), as well as learning activities (second column). The overall educative style GLS-weighted index (third column) shows a sizable and significant effect (across the three inference procedures) of the _Plus_ modality, with an increase of 0.49 standard deviations in the promotion of educative parenting styles to parents during the home visits. Other aspects of the parent-child relationship that are focused on emotional practices do not seem to systematically vary across the two program modalities.

The evidence presented in this section points toward cross-modality variation in the quality of both the parent/mentor interactions and parent/child interactions as a potential mechanism behind the observed difference in children’s outcomes. While we cannot separately quantify 17
Table 4: The Role of Mentors in Fostering Parental Attitudes—Second Experiment

||Panel A: Paren<br>Quantity|ts and Mento<br> (Last 60 Days|rs Interactio<br>)|ns (as reported b|y the parents)<br>Quality|||
|---|---|---|---|---|---|---|---|
||Meetings|Visits|Index|Inform|Advise|Index||
|||||About Child|About Child|||
|API Plus|1.039|0.726|0.362|0.102|0.100|0.251||
||[0.147]|[0.125]|[0.062]|[0.057]|[0.034]|[0.040]||
||_{_0.194_}_|_{_0.171_}_|_{_0.094_}_|_{_0.097_}_|_{_0.056_}_|_{_0.070_}_||
||(0.194)|(0.194)|(0.100)|(0.078)|(0.078)|(0.078)||
|Observations|482|491|504|354|353|357||
|Clusters|123|124|124|113|112|113||
||Panel B: Pa<br>Educ|renting Styles<br>ative Style|that Are P|romoted by the|Mentors (as reporte<br>Emotional St|d by the mento<br>yle|rs)|
||Communication|Learning|Index|Share|Self-Knowledge|Manage|Index|
|||||Feelings||Transitions||
|API Plus|0.178|0.168|0.494|0.049|0.030|0.142|0.194|
||[0.038]|[0.077]|[0.018]|[0.627]|[0.756]|[0.123]|[0.312]|
||_{_0.043_}_|_{_0.091_}_|_{_0.029_}_|_{_0.635_}_|_{_0.753_}_|_{_0.134_}_|_{_0.321_}_|
||(0.074)|(0.075)|(0.043)|(0.843)|(0.843)|(0.308)|(0.558)|
|Observations|107|107|107|107|107|107|107|

_Notes_ : This table shows OLS estimates and the associated _p_ -values of the API _Plus_ modality on survey-based measures of interactions between parents and mentors (Panel A) and the different parenting styles that are promoted by the mentors during their interactions with the parents. For a detailed description of the outcome variables used in this table, see Appendix A.2. _p_ -values reported in brackets refer to the conventional asymptotic inference. _p_ -values reported in braces are computed using randomization inference (randomization- _t_ ). All _p_ -values account for clustering at the school level. _p_ -values reported in parentheses are adjusted for testing the effect of API _Plus_ for the different families of outcomes (quantity and quality of interactions, parenting styles) through the stepwise procedure described in Romano and Wolf (2005a,b, 2016).

the relative contribution of each additional training module, the increase in the quality of the home visits is likely to originate from the mentors’ peer-to-peer sessions, which had the exact role of helping the mentors with enhanced training to communicate more effectively with parents. As mentioned in Section 2.2, these workshops enable interactions and information sharing among the participants, while the extra week of initial training is instead focused on pedagogical practices targeted to children at school. Qualitative evidence seems indeed to corroborate this hypothesis, as summarized by the following quotes from mentors who have participated in the peer-to-peer meetings (see Appendix A.3 for more details):

- _“During the workshops I was told that I should be able to adapt to the context of the community and understand the local living arrangements in order to establish a dialog with the parents without modifying what they conceive as their environment.”_

- _“It was recommended that we pay frequent home visits so as to establish a relationship with the parents and gain their trust.”_

- _“[The workshops] exposed us to effective strategies of other mentors [for_

18
_dealing with parents] that we could try and implement in our community.”_

# **4 Threats to Scalability**

Over the summer of 2016, after learning about the results of the second experiment, the government decided to replace the _Original_ program with the enhanced training modality. All its primary schools, including the 224 schools that were part of the evaluation sample, were deemed eligible to receive the _Plus_ program modality. The overall scale of the operation of the mentoring intervention—including the total number of mentors that participated in the program—remained constant in the periods before, during, and after the experiments. This single policy change creates two interesting circumstances that are informative for our case study on scaling. On the one hand, schools that were part of our second experiment experienced a change in the situation—from the research setting to the government implementation. On the other hand, the rest of the schools, that were not part of the experiment but received the mentoring program under the _Original_ modality, underwent a reform in program design within the same government situation. In this section and the next one, we focus on the sample of the experimental schools in order to zoom into the threats and mechanisms of scaling brought about by the new situation. In Section 6, we discuss policy impacts on education outcomes for experimental schools as well as for the overall population of schools in the state of Chiapas.

To study the scale-up problem in our context, we analyze three key aspects outlined in Al-Ubaydli et al. (2020), namely inference, representativeness of the population, and representativeness of the situation. Although it will not be part of our discussion, we also want to mention additional features of the experimental design that may speak to other threats to scaling. First, the relatively large units of randomization (school community) are robust to local general equilibrium/spillover effects, which are often relevant in field experiments that are implemented at scale. Second, our field experiment has been implemented by the research team in close collaboration with the government agency that was later in charge of its policy implementation. Hence, the design of the two program modalities bears in mind the supply-side considerations of scaling, as well as various financial and local institutional constraints.

19
Table 5: Joint Test of Significance Within and Across Experiments ( _p_ -values)

||First Experiment|Second Experiment|Both Experiments|
|---|---|---|---|
|Api Original = Control|0.828|0.411|0.707|
|Api Plus = Control|.|0.001|.|
|API Original = API Plus|0.114|0.001|0.002|

_Notes_ : This table reports randomization-inference (Randomization-t) _p_ -values for the omnibus test of overall experimental significance of each separate hypothesis (Young, 2019). An asymptotic _p_ -value is reported for the hypothesis that API Original = API Plus in the first column, which is tested across experiments. All _p_ -values account for clustering at the school level.

# **4.1 Inference**

Inference problems arise when researchers and practitioners want to learn to what extent existing evidence advocates for policy decisions. We focus on whether (i) the lack of effectiveness of the _Original_ modality is indicative of a null result, and (ii) the large impact of the _Plus_ modality on schooling outcomes for children and engagement outcomes for parents is a false positive.” We jointly consider two key outcomes—the overall index of student achievement and the overall index of parental engagement—and compute _p_ -values of overall statistical significance (Westfall and Young, 1993). Following the insights in Maniadis et al. (2014), we harness the value of the two experiments to bolster the credibility of our empirical evidence. We test hypotheses across experiments using Fisher’s combined probability test, which is akin to the joint statistical significance test usually invoked in meta-analyses.<sup>12</sup> Table 5 shows the results. Consistently within and across experiments, the _Original_ modality does not generate actionable evidence (and yet, the government implemented such program modality at scale). The evidence displayed in the first row documents a lack of significance of such variant of the mentoring program on children’s achievements and parental investment. The results in the second and third rows of Table 5 document a highly significant impact of the mentoring program under the _Plus_ modality, both when compared to the control group with no mentors ( _p_ -value = 0.001) and the _Original_ modality. The relatively noisier estimates of the first experiment do not allow us to reject the hypothesis of equal treatment effects between the two modalities when tested across experiments ( _p_ -value = 0.114). This result highlights the importance of the design of the second experiment, in which we replicate the

> 12 The Westfall-Young procedure uses the joint distribution of _p_ -values across all equations so as to minimize the loss of power brought about by the multiple testing adjustment within a given experiment. Combined _p_ -values across experiments are obtained using Fisher’s formula: _−_ 2<sup>�</sup><sup>_k_</sup> _i_ =1<sup>_log_(</sup><sup>_pi_)</sup><sup>_∼χ_</sup> 2<sup>2</sup> _k_<sup>,where</sup><sup>_pi∼U_[0</sup><sup>_,_1]</sup> is the _p_ -value for the _i_<sup>_th_</sup> hypothesis test and _k_ = 2 is the number of independent experiments being combined.

20
_Original_ modality along with the new _Plus_ modality. In the second experiment, we strongly reject that the _Plus_ modality is equally effective to the _Original_ modality ( _p_ -value = 0.001). A very similar result holds through a combined probability test across both experiments ( _p_ -value = 0.002), which is reported in the third column of Table 5.

The joint inference drawn from the two experiments seems to convincingly point toward the relative effectiveness of the _Plus_ modality when compared to both the _Original_ modality and the control group with no mentors. Given that the policy reform under study is a change from the _Original_ to the _Plus_ modality, we discard the first experiment in the rest of this section and focus our analysis on the schools that participated in the second experiment.

# **4.2 Representativeness of the Population**

Research findings from field experiments may sometimes be difficult to generalize because, in the language of Al-Ubaydli et al. (2020), the properties of the study population may differ from the population of interest to policy makers. Heckman (1992) discusses selection into field experiments and finds that the characteristics of subjects who participate can be distinctly different from those of subjects who do not participate. In Table 6 we compare means in observable characteristics between our experimental sample and the overall population of schools in the state of Chiapas. The descriptive statistics for the sample of experimental schools are shown in column two, and they appear remarkably balanced when compared to the respective statistics in the overall population that are displayed in the first column. As shown in the third column, we cannot reject equal means across the several variables assessed. There is a very small imbalance in the number of local instructors, which is only marginally significant.

We next study whether the average impacts of the _Plus_ modality in the second experiment have ex-ante external validity with respect to the impact of the mentoring intervention at scale in the broader population of schools in Chiapas. To do this, we evaluate whether program impacts vary along the program eligibility criteria used by the government during the policy implementation (see Section 2.1). The idea behind this exercise is that any variation in treatment effects along those dimensions may be indicative of the extent to which program effects may change because of the underlying differences across populations. Table 7 displays heterogeneous treatment effects of both _Original_ and _Plus_ modalities along two criteria that are time invariant and hence plausibly unaffected by the intervention: whether the community where the school is located is categorized as having high or very 21
Table 6: Differences Across Populations

||All Chiapas<br>Mean (SD)|Second Experiment<br>Chiapas vs. Second Experiment<br>Mean (SD)<br>_p_-value|
|---|---|---|
|||Panel A: Community Characteristics|
|Number of households|34.625|29.329<br>0.486|
||(109.863)|(50.234)|
|Total population|140.293|121.389<br>0.494|
||(394.371)|(240.562)|
|Share economically active|0.298|0.303<br>0.361|
||(0.073)|(0.070)|
|Water connection (Y/N)|0.033|0.023<br>0.454|
||(0.178)|(0.151)|
|Sewer system (Y/N)|0.018|0.009<br>0.346|
||(0.134)|(0.096)|
|Share of illiterates|0.268|0.270<br>0.832|
||(0.175)|(0.167)|
|Share of dwellings with dirt foor|0.328|0.363<br>0.126|
||(0.319)|(0.322)|
|Garbage collection (Y/N)|0.025<br>(0.158)|0.023<br>0.842<br>(0.151)|
|||Panel B: School Characteristics|
|Average test score (Spanish) 2010|425.173|431.340<br>0.158|
||(57.245)|(60.810)|
|Average test score (Math)|415.998|421.333<br>0.363|
||(76.967)|(80.895)|
|Number of students|14.023|14.770<br>0.205|
||(8.403)|(7.069)|
|Number of local instructors|1.216<br>(0.449)|1.279<br>0.054<br>(0.514)|
|Share students over-age|3.125|3.620<br>0.264|
||(6.285)|(5.629)|
|Observations|1,475|230|

_Notes_ : Means and standard deviations in parentheses for various characteristics collected before the introduction of the API program. The last column shows asymptotic _p_ -values for mean differences between the overall population and the experimental sample. Panel A shows community-level characteristics from the population census (2010), whereas Panel B displays school-level variables from the school census (2010). See Appendix A.1 for more details on the data sources.

high “marginality,” as defined according to the National Population Council or CONAPO (Poverty 1), and whether the community was targeted by an anti-poverty program (Poverty 2).<sup>13</sup> In the sample of schools in the second experiment, approximately one-third satisfy the Poverty 1 criterion, 70 percent satisfy the Poverty 2 criterion, and 25 percent satisfy both criteria. We run separate regression models for three summary outcomes of the intervention on both students and parents: the overall index of student achievement, the indicator for enrollment in seventh grade, and the overall index of parental engagement. Estimation

> 13For details on the Poverty 1 index, refer to `https://www.gob.mx/cms/uploads/attachment/file/ 685308/Nota_t_cnica_IML_2020.pdf` , accessed on August 2022.

22
Table 7: Heterogeneity in the Impact of the Program by Eligibility Criteria

||Childre|n’s Outcomes|Parental Outcome|
|---|---|---|---|
||Overall Score|Enrolled Secondary|Engagement Index|
|API Original|0.088|0.134|-0.018|
||[0.419]|[0.205]|[0.887]|
|API Original_×_ Poverty 1|0.090|-0.104|-0.046|
||[0.637]|[0.420]|[0.816]|
|API Original_×_ Poverty 2|0.091|-0.033|-0.009|
||[0.490]|[0.767]|[0.949]|
|API Plus|0.320|0.273|0.464|
||[0.047]|[0.024]|[0.000]|
|API Plus_×_ Poverty 1|0.094|0.015|0.100|
||[0.638]|[0.902]|[0.643]|
|API Plus_×_ Poverty 2|0.010|-0.216|-0.173|
||[0.963]|[0.128]|[0.386]|
|Original(Pov. 1)=Original(Pov. 2)|[0.995]|[0.681]|[0.873]|
|Plus(Pov. 1)=Plus(Pov. 2)|[0.816]|[0.297]|[0.444]|
|Number of clusters|224|182|224|
|Observations|1045|468|1045|

_Notes_ : This table shows OLS estimates and the associated _p_ -values (in brackets) on student and parental outcomes measured after two academic years of exposure to the API program under the second experiment designed and implemented by the authors in collaboration with the government. For a detailed descriptions of the test scores used in this table, see Appendix A.2. The dependent variables in the first and third columns are standardized with respect to their means and the standard deviations in the control group. The dependent variable in the second column is computed from administrative school records (see Appendix A.1). All _p_ -values account for clustering at the school level.

results reveal limited variation in program impacts along both poverty measures, with effect sizes for the interaction terms with the indicator variables for the program modalities that are not statistically different from zero, and not statistically different from each other.

Taken together, the evidence shown in this section documents that differences across populations (if any) are unlikely to represent a meaningful threat to scalability in this context. There is a very high degree of similarity in observable characteristics between the experimental sample and the overall population of schools in Chiapas. Program impacts are also not heterogeneous along the determinants of the rollout of the program under the government implementation, which is indicative of the fact that the community/school targeting process is unlikely to play a role for the scalability of the program.

23
# **4.3 Possible Threats from the New Situation**

Notwithstanding sizable and significant effects on a sample that is representative of the population of interest, the success of the mentoring intervention at scale is not guaranteed. The difference between the implementation protocol in our research setting—where we had an active role in guaranteeing the smooth progress of the field experiment—and the government operations can translate into contrasting aftermaths of the program. For example, the criteria for closing schools were very different in the two situations. Although the official enrollment threshold for closing schools is six students, schools in the second experiment were allowed to remain open if they had at least three enrolled students in either of the two school years when the experiment took place. In addition, children in schools with more than 29 enrolled students were required to transfer to schools in the regular public school system. As a result, only two schools closed in the sample of 230 schools in the second experiment (see Section 2.2).<sup>14</sup> Hence, a threat to the success of the program in the new policy situation may come from the possible school closures as a result of the mentoring intervention under the government implementation. If instead the presence of a mentor increases the probability that schools remain open, this mechanism could represent an opportunity to learn something about the mechanisms behind the scalability of the program in our context. Figure 1 shows the relationship between school closures, as measured by the year-to-year presence in the school census two years after the second experiment, and school size, as measured by the number of enrolled students in the school year before the second experiment. The green (lighter) line shows how the probability of school closures varies by school size for schools that did not receive the program during the government intervention. The probability of school closure is bimodal, and its two modes are positioned around the two critical enrollment thresholds.

> 14While school closures are obviously a “non-negotiable” aspect (List, 2022) for the success of the API program, there may be other “negotiable” differences in the program implementation across the experimental and the policy settings that we cannot directly study due to a lack of monitoring data outside of the experimental sample/period. First, to avoid refusal of the assigned mentor among the communities of the evaluation schools, each mentor in the experimental sample was provided with two baskets of food, throughout the school year, as donations to the community leaders as well as for personal consumption. Second, as a way to attenuate the potentially detrimental consequences of mentors’ dropping out of the program during the evaluation period, the government delegates in Chiapas arranged for a replacement within two weeks from the day of a mentor’s departure from a community. If the dropout was part of the _Plus_ group, the replacement would receive an additional three-day training session that would make up for the content covered during the extra week of the initial training session. Third, there might be differences between the experiment and the policy rollout in the implementation of the training module of the _Plus_ modality, such as the number of extra days and the content of the curriculum of the initial training as well as the frequency of the peer-to-peer sessions.

24
<!-- Start of picture text --> .||S<br>||<br>I |<br>no} I |<br>2)B te)2. | | ©Ss.<br>2I |<br>We) I |<br>2 a 5 = ! | | | 3 . =>c<br>= a<br>>I|<br>=| |<br>a te) I I N<br>ae! Sy<br>2 | |<br>oO I |<br>||<br>||<br>°o ! °<br>0 10 20 30 40<br>Number of Enrolled Students<br>——— Closures (Non-API) --———_ Closures (API) Density<br><!-- End of picture text -->
in schools with more than 29 students to transfer to schools that are part of the regular public school system. Given the evidence on the effect of the mentoring intervention on parental engagement (see Section 3), in this section we analyze the role of parents, and more broadly, of the community-level parental engagement, as a potential mechanism behind the scalability of the program. We first lay out a simple model of skill formation and parental investment, where the individual incentives for parents to invest in educational activities are jointly determined at the community level. The model provides us with a framework to study the threat to scalability from changes in situations. We then document empirical evidence that is consistent with the key model predictions and that is difficult to reconcile with alternative, more direct, channels of influence of the mentors on school closures.

# **5.1 Theory**

There are various local communities ( _c_ ), each composed of a number of families _Nc_ . Each family _i ∈{_ 1 _, . . . , Nc}_ decides whether to engage in parenting, _Ii ∈{_ 0 _,_ 1 _}_ . The returns to parenting are defined by the following technology of skill formation: where _θi_ is a child’s skills, while _Nc_ <u>1</u> _−_ 1 � _j_ = _i_<sup>_Ij_representstheaverageparentalengagement</sup> among other parents in the same community. The parameter _φ_ characterizes the degree of substitutability between individual parental engagement and community-level parental engagement in the process of child development. Parameter _A_ is total factor productivity, while _θi,_ 0 represents the child’s initial skills.<sup>15</sup> We model parents as paternalistic over their children’s skills, and we assume that there is a cost of parenting so that the parental utility function is _μ_ ( _Ii_ ) : _Ui_ = _−μ_ ( _Ii_ ) + _θi_ . The parental decision problem is defined as follows:

> 15We omit in equation (1) the shares of the parental investment since the total factor productivity term, _A_ , allows us to appropriately re-scale the constant elasticity of substitution function without loss of generality.

26
In this framework, the incentives for parents to engage with the education of their children depend upon the community-average parental engagement as well as on the productivity of those investments. An equilibrium in this economy is defined as the optimal choices of families ( _Ii_<sup>_∗_) that are consistent with the endogenously determined community-level engagement:</sup> � _Ii_<sup>_∗_(</sup><sup>_I_</sup> _−_<sup>_∗_</sup> _i_<sup>)</sup> � _Ni_ =1 _c_<sup>.</sup>

# **Proposition 1** _This economy exhibits two types of equilibria:_

_1. Free Riding Equilibrium: parents free ride on each other, and in equilibrium there is no community engagement (Ii_<sup>_∗_= 0</sup><sup>_∀i)._</sup> _2. Collaborative Equilibrium: all parents in the community are engaged in the process of children’s learning (Ii_<sup>_∗_= 1</sup><sup>_∀i)._</sup> We introduce in this framework an educational intervention _τ ∈{_ 0 _,_ 1 _}_ —such as the API program—that directly affects both the process of skill formation of children and the incentives of parents as follows: The impact of the intervention not only hinges on the direct effect on children’s skill _γ_ 0, but also on its ability to shift the incentives of parents through higher productivity of investment ( _γ_ 1), which in turn affects the community-level parental engagement. The total effect on children’s outcomes ∆ _θi_ = _θi_ ( _τ_ = 1) _− θi_ ( _τ_ = 0) can be written as follows: We define a threat to scalability in this framework as a deterioration of the direct impact of the program. More precisely, for two given situations _s_ and _s_<sup>_′_</sup> —such as the field experiment and the government implementation in our context—we posit that _γ_ 0( _s_<sup>_′_</sup> ) _< γ_ 0( _s_ ). The extent to which this translates into a deterioration of the overall effectiveness of the program depends on the endogenous response of parents. In one scenario, the change in the situation induces the program to fail at scale because of the lack of parental responses. In the second scenario, the program is able to promote coordination and engagement of parents in the local 27
Figure 2: Pathways To Scale <!-- Start of picture text --> 2<br>1.9<br>1.8<br>1.7<br>1.6<br>1.5<br>1.4<br>1.3<br>1.2<br>1.1<br>1<br>0.3 0.4 0.5 0.6 0.7 0.8 0.9 1<br>Incentive to Free Ride ( )<br>)<br>1<br>Minimum Treatment Effect for Scalability (<br><!-- End of picture text --> _Notes_ : The figure shows the relationship between the incentive to free ride among parents ( _φ_ ) and the program’s impact on the productivity of parental investment as predicted by a calibrated version of the model ( _μ_ ( _Ii_ ) = _−_ 2 _· Ii_ , _Nc_ = 1000).

community, therefore possibly offsetting the threat to scalability with respect to changes in the situations.

The parameter _φ_ , which captures the degree of complementarity between parents in the technology of skill formation of children, lies at the core of the equilibrium selection in the model (Free Riding vs. Collaborative, see Proposition 1). This parameter pins down the incentives for parental cooperation in the local educational activities. Low values of _φ_ represent high incentive for cooperation, while positive levels of _φ_ induce free-riding among parents. Figure 2 shows how the impact of a program at scale across different situations depends upon the degree of parental collaboration ( _φ_ ) and its ability to trigger local spillover effects. The figure shows that in economies with a high degree of parental collaboration (small _φ_ ), a relatively small impact on the productivity of investment ( _γ_ 1) can induce a shift toward the Collaborative Equilibrium. Economies with a lower degree of parental cooperation (high _φ_ ) need larger impacts of the program on the productivity of parental investment to trigger the social determination of human capital investment at the community level.

28
# **5.2 Evidence on Parents as Means of Scalability**

We test the model’s key prediction using variations across situations (the first experiment was run by the government, while the second experiment was run by the research team) and across program modalities (API _Original_ and API _Plus_ ). In particular, we study whether a different response of parents in terms of their engagement both at the local school and with their children (see Table 3) subsequently triggers a differential impact of the mentoring intervention on school closures. School closures, which in our context represent a major disruption to the program’s continuity and effectiveness, should be interpreted as an outcome of the deterioration effect of the program once it is implemented at scale.

The first two columns of Table 8 show the reduced-form effects of the two randomized program modalities—in both the first experiment (first column) and the second experiment (second column)—on the probability that schools close in the second year of the national rollout of both programs. The _Original_ modality displays small and noisy effects on school closures in both experiments, which are not statistically different from zero. This result suggests that situations that do not promote parental engagement do not differ from the status quo rates of school closures, which range between 5 percent (first experiment) and 8 percent (second experiment) in the experimental control groups with no mentors.

The second column of Table 8 shows that the _Plus_ modality, as Table 3 shows significantly boosts parental engagement, has a significant impact on school closures. Schools that were assigned to the enhanced modality during the second experiment experience are less likely to close permanently ( _−_ 8.3 percentage points) two years after the _Plus_ modality was adopted by the government, which is statistically different from zero at the 95 percent confidence level. This result echoes previous evidence on the relationship between the probability of closures for schools that receive a mentor during the government implementation of the _Plus_ modality, which is shown in Figure 1. Given that the probability of receiving a mentor during the government implementation is orthogonal with respect to the randomized API assignment of the second experiment, this evidence rules out channels other than parental engagement through which the presence of the mentors can keep the schools open.<sup>16</sup> The IV estimates shown in the third column of Table 8 go a step further and quantify the extent to which parental engagement affects the probability of school closures. Because of

> 16Approximately half of the schools in any of the treatment arms and the control group of the second experiment received a mentor by the second year of the national rollout of the _Plus_ modality. This share is balanced across treatment arms after controlling for the program eligibility criteria (see Section 2.1): _p_ -value( _Original_ ) = 0.367, _p_ -value( _Plus_ ) = 0.660.

29
Table 8: School Closures and Parental Engagement

|||Outcome: School Clos|ures|
|---|---|---|---|
||First Experiment|Second Experiment|Second Experiment, IV|
|API Original|0.031<br>[0.549]|-0.031<br>[0.396]|-0.031<br>[0.410]|
|API Plus||-0.083<br>[0.030]||
|Overall Parental Engagement|||-0.217<br>[0.021]|
|Observations|80|224|1045|
|Clusters|.|.|224|
|F-Stat (Excl. Instruments)|||13.833|

_Notes_ : This table reports the estimates for the reduced-form effects of the API modalities during the two experiments (columns 1 and 2) on the probability of school closures, as well as the instrumental variable estimates of the impact of parental engagement on school closures. In the third column, the randomized API _Plus_ modality during the second experiment is used as an instrumental variable, while the randomized API _Original_ modality is included as a control variable. The dependent variable is an indicator variable for whether the school is closed in the fall of 2014 (column 1) or in the fall of 2018 (columns 2 and 3). The variable “Overall Parental Engagement” is the same variable used in the last column of Table 3. All _p_ -values account for clustering at the school level. _p_ -values reported in brackets account for clustering at the school level.

the contextual information on the role of the parental association in deciding school closures discussed previously in this section, we posit that parents are the main channel through which the _Plus_ modality of the API program affects school closures. The differential impacts of the two program modalities on both parental investment and school closures shown in the first two columns of Table 8 are consistent with this exclusion restriction. We find that an increase of half a standard deviation in the overall parental engagement index is causally associated with a reduction of 11 percentage points in the probability that their children experience a school closure. This effect is both statistically and quantitatively significant.

We complement these findings with qualitative evidence on the role of parents in ensuring continuity in schooling activities (see Appendix A.3). As reported by the community instructors, parents may have more at stake in keeping the schools open as they invest in durable goods for the local school:

- _“[Parents] help manage the school and contribute by improving the fencing, painting the walls, fixing the toilets, as well as buying school materials.”_

- _“[Parents] serve the needs of the school with construction works and they provide food to the local instructor.”_

30
Table 9: Parental Investment by Proxies of Community-Level Collaboration

||Engage at School|Manage School|Engage With Child|Engagement Index|
|---|---|---|---|---|
|API Plus_×_ No Confict|0.161<br>[0.176]|0.159<br>[0.103]|0.370<br>[0.000]|0.364<br>[0.001]|
|API Plus_×_ Confict|0.926<br>[0.000]|0.281<br>[0.183]|0.592<br>[0.005]|0.914<br>[0.000]|
|Confict=No Confict|[0.000]|[0.596]|[0.329]|[0.020]|
|Mean Control (Confict)|-0.006|-0.059|0.152|0.011|
|Mean Control (No Confict)|-0.023|0.013|-0.013|0.003|
|Number of clusters|224|224|224|224|
|Observations|1045|1045|1045|1045|

_Notes_ : This table shows OLS estimates on parental outcomes measured after two academic years of exposure to the API program under the second experiment designed and implemented by the authors in collaboration with the government. The variable Conflict takes the value of one if least on hostile event related to land property, religion, elections, crime, or drug addiction is reported at the locality level in the population census (2010). For a detailed descriptions of the variables used in this table, see Appendices A.1–A.2. The dependent variables are standardized with respect to their means and the standard deviations in the control group. All _p_ -values account for clustering at the school level. Asymptotic _p_ -values reported in brackets are clustered at the school level.

As reported by the mentors, parents follow up with their children on homework and other pedagogical material whenever the mentor is busy attending tasks outside of the community: “ _Parents used to provide support with homework whenever mentors are visiting other communities ensuring pedagogical support, so that upon the return of the mentors they are able to make progress in the schooling activities without setbacks._ ” We next empirically investigate the second prediction of the model, as depicted in Figure 2 and discussed above. We take advantage of information from the 2010 locality-level census on the degree of social hostility in the community (see Appendix A.1), which is based on hostile event related to land property, religion, elections, crime, or drug addiction. This measure proxies the degree of collaboration in the community. In the 224 communities that form part of the second experiment, we construct an indicator variable for the presence of a conflict in the community if at least one of these hostile events is reported. We then interact this variable with the experimental API _Plus_ assignment and regress the various survey-based measures of parental engagement collected during the second experiment on these interaction terms.

31
Table 9 displays the estimation results. In line with the prediction from the model, communities with higher hostility display a higher parental response to the _Plus_ program when compared to communities with no conflicts, as parents need to overcome the higher incentive to free ride.<sup>17</sup> The impacts on parental engagement at school are shown in the first column, and it is approximately eight times larger in communities with conflicts than in communities without conflicts. The impacts are twice as large when considering activities related to managing school resources and engaging directly with children in educational activities, although in this case they are not statistically different from the corresponding effects without conflicts (second and third columns). In the last column we look at the overall parental engagement index. Communities with conflicts exhibit impacts on parental behavior (+0.91 standard deviations) that are 2.5 times larger than communities with no conflicts (+0.36 standard deviations)—this difference is statistically significant at the 95 percent confidence level.

Previous literature has highlighted how parental investments and parenting styles are responsive to the environments that families face (Doepke and Zilibotti, 2017; Agostinelli, 2018; Agostinelli et al., 2020). Our results shed light on how the success of an educational program depends upon the local engagement of parents in educational activities and how its scalability is effectively a socially determined outcome.

# **6 Policy Impacts at Scale**

In this last part of the analysis, we discuss the impacts of the government-run program on various educational outcomes. After 2016, the government fully converted the mentoring program into the _Plus_ modality, and all schools were in principle eligible to receive the mentors. Schools were assigned a score between one and four, with one denoting the highest priority level. The scores are based on a combination of criteria that includes school performance in the national learning assessment, whether the school has six or more primary students enrolled, whether the school received the API program in the period between 2009 and 2015, the level of marginalization of the community where the school was based, and whether the community was targeted by an anti-poverty program (see Section 2.1). The size of the new program did not change relative to the previous implementation of the _Original_ modality, including the number of available mentors. In the fall of 2016 there were 535

> 17Notice that few to no school closures were detected in villages that were part of the _Plus_ modality, independently of whether the community experienced any conflict.

32
mentors in Chiapas, and given these constraints, mentors are allocated across communities on a rotating basis.

# **6.1 The Exposure Effect of the Program**

We start by analyzing the initial transition of the schools in the evaluation sample from the experimental situation to the policy at scale. We do this by estimating the following regression model: where 1 � _ExpPlusj_ ( _i_ ) = _k_ � _∈{_ 0 _,_ 1 _}_ represents an indicator variable for whether school _j_ is exposed to _k ∈{_ 0 _,_ 1 _, . . . , K}_ years of API _Plus_ program, while **_X_** _j_<sup>_Criteria_</sup> ( _i_ ) is a vector of indicator variables for the program eligibility criteria. Our outcome of interest ( _Yj_ ) represents the 2016–2017 school-level transition rates to seventh grade. This variable is constructed from the same administrative source we used for our experimental evaluation. The sample includes 207 schools of the 224 that were part of the experiment. Beyond a school that permanently closed, the sample attrition is caused by schools not having sixth graders during that school year. This fact is consistent with the multi-grade nature of the CONAFE system. Attrition is balanced with respect to the total years of exposure to API _Plus_ ( _p_ -values = 0.467, 0.812, and 0.568).

We exploit the change in situation to analyze the exposure effect to the program one year after the government rollout, when school closures were still minimal (only one school closed among our 224 schools by the fall of 2017). After the first year of the rollout, schools started to close more intensively (11 additional schools closed during the second year); hence, outcome variables based on survey or administrative school data during this period would suffer from endogenous censoring.<sup>18</sup> By the spring of 2017, the total years of exposure to API _Plus_ range from zero to three depending on whether a school received the _Plus_ modality for two years during the second experiment, as well as on the API _Plus_ government assignment in the year after the experiment. The underlying assumption of this approach is that, once controlling for the official government criteria, the remaining variation in the program’s

> 18For this reason, in Section 6.2 we take advantage of the 2019 Census information that is not subject to this issue. The Mexican Census is available only every ten years, and hence we cannot use this information to estimate the model in equation (4).

33
Figure 3: The New Situation in the Experimental Sample of Schools <!-- Start of picture text --> Urban Secondary School Enrollment<br>1 Year 2 Years 3 Years<br>Total Years of Exposure to API Plus<br>Point Estimate 90% CI 95% CI<br>.5<br>.4<br>.3<br>.2<br>.1<br>0<br>Treat Effect on 7th Grade Enrollment<br>−.1<br><!-- End of picture text --> _Notes_ : This figure shows OLS estimates of the years of exposure to the mentoring program on the probability of enrolling in seventh grade during the transition from the second experiment to the government implementation of the _Plus_ modality. Vertical lines overlaid on each bar display the 95 percent and 90 percent confidence intervals, respectively. Confidence intervals are based on asymptotic inference.

assignment across localities in the year after the experiment is as good as random: While this assumption is obviously not testable, we examine whether the API _Plus_ assignment is conditionally balanced with respect to predetermined educational outcomes. Table B-12 shows the results of this placebo test. The outcomes we use are the 2013 scores in the national standardized test (Spanish, math, and science, see Appendix A.1).<sup>19</sup> The estimated coefficients of the years of exposure to API _Plus_ are not statistically different from zero. The point estimates are relatively small, especially after controlling for the assignment criteria, and their signs do not suggest any pre-existing positive trend. For all these reasons, we believe this evidence provides some support to the plausibility of (5) in our setting.

Figure 3 plots the estimated _β_ 1 _,k_ coefficients shown in equation (4), where zero years of exposure represents the reference category. The results show a positive exposure effect to the program one year after the government rollout. The effect of the _Plus_ modality goes from 3 percentage points after one year to more than 35 percentage points after three years

> 19The year 2013 is the last year in which the national standardized test was applied universally in Mexico.

34
of exposure. The average effect of three years of exposure on the probability of enrolling in seventh grade is very large and precisely estimated ( _p_ -value _<_ 0.001). The magnitude of this effect implies that the enrollment rates in these disadvantaged and rural areas achieve the secondary school enrollment rates in urban Mexico (95 percent). The average marginal effect of an extra year of the program is +10 percentage points ( _p_ -value = 0.006) in the probability of enrolling in seventh grade. The effect of two years of exposure, although not statistically different, is larger than the experimental estimate displayed in Table 2 (fifth column) suggesting that the impact does not fade out after one year.

# **6.2 The Effect of the Program Beyond the Evaluation Sample**

We now broaden our analysis to the entire population of schools in the state of Chiapas. We examine whether the _Plus_ modality of the program at scale has promoted educational opportunities for children in these disadvantaged communities, possibly resembling the results from the second field experiment (see Section 3). To do so, we match administrative records on the government rollout of the program during the fall of 2017 with village-level educational outcomes from the population census data (data collection in the fall of 2019) for the majority of the schools and communities, which include those that were part of the second experiment.<sup>20</sup> Our first outcome is the village-level lower-secondary enrollment rates among children between 12 and 14 years old. This variable is available in the census for 1,417 communities in Chiapas. It is not immediately comparable with our previous measure of enrollment in seventh grade for two reasons. First, the census-based information represents the stock (rates) of children enrolled in secondary school in a given year, while our previous measure represents the flow of new students enrolling in secondary schools. Second, the census-based variable includes children in the village who are enrolled in primary schools that are not eligible for the API program, which converts the analysis of the program at scale into an intent-to-treat analysis. Another educational outcome from the population census that we use is the rate of child literacy for children between eight and fourteen years old, which represents an available measure of children’s achievement. This variable is available

> 20The match between the universe of schools and the localities of the population Census is one to one, as each village has at most only one primary school. The coverage of the census data is not universal, and we were not able to match nearly one-third of the 2,063 schools in Chiapas in 2019. For both educational outcomes in the census data we cannot reject the hypothesis that the probability of missing observations is balanced with respect to the program assignment ( _p_ -values for secondary school enrollment and child literacy are 0.728 and 0.430, respectively). For further details on the census sampling design, please refer to: `https://www.inegi.org.mx/contenidos/productos/prod_serv/contenidos/espanol/bvinegi/ productos/nueva_estruc/702825197629.pdf` , accessed on August, 2022.

35
in the census for 1,440 communities in Chiapas. Finally, we match information about school closures from the school census in the same year (2019).

This set of educational outcomes is particularly conducive for our analysis. Secondary school is a critical period for the educational outcomes of the disadvantaged population under study, as more then a quarter of the 12 to 14 year olds in Chiapas are out of school. Likewise, 13 percent of school-aged children are still illiterate. The year of the data collection in the census (2019, two full school years after the rollout of the _Plus_ modality at scale) is consistent with the length of exposure to the API program in the second experiment. These census-based outcomes cover the quasi-universe of the localities in Mexico and, unlike other survey-based or administrative test score measures, they are not subject to any censoring during the data collection due to school closures. This allows us to avoid the concerns about selection bias due to differential school closures induced by the program at scale (see Section 4.3).

We analyze the impact of the policy implementation of the program using the following linear regression model: where _Yj_ is a school-level outcome for school _j_ , while _Plusj_ takes a value of one if school _j_ receives a mentor during the government implementation of the _Plus_ modality, and zero otherwise. The vector **_X_** _j_<sup>_Criteria_</sup> consists of all the criteria used for the assignment of the program. The parameter of interest, _α_ 1, represents the effect of the program during the government implementation on the outcome of interest. As was the case for the exposure analysis discussed in the previous section and formalized by equations (4)-(5), to causally interpret the estimates we need the assignment of the program across communities to be _conditionally_ as good as random. In other words, conditional on the assignment criteria, schools that receive and do not receive the program at scale are similar in terms of unobserved characteristics. As before, we run some placebo tests to bolster the credibility of this identification assumption in our setting. Table B-13 shows the results. The 2017 government assignment is not unconditionally random (odd columns of the table), as priority is given to more disadvantaged communities. Instead, when we control for the vector **_X_** _j_<sup>_Criteria_</sup> , the estimated coefficients displayed in the even columns of Table B-13 are very small and statistically insignificant.

Panel A of Figure 4 shows the results for secondary school enrollment after two years from 36
Figure 4: Policy Impact on Education Outcomes <!-- Start of picture text --> Non−Experimental Schools Experimental Schools All Schools Non−Experimental Schools Experimental Schools All Schools<br>.08<br>.15<br>.06<br>.1<br>.04<br>.05 .02<br>Treat Effect on Rates of Child Literacy<br>Treat Effect on Secondary School Enrollment 0<br>0<br><!-- End of picture text --> Panel A: Secondary School Enrollment Panel B: Child Literacy _Notes_ : The bars in the figure represent the OLS estimates of the assignment to the API program during the government implementation of the _Plus_ modality on school-level secondary school enrollment rates (Panel A) and child literacy rates (Panel B). Vertical lines overlaid on each bar displays the 95 percent and 90 percent confidence intervals, respectively. Confidence intervals are based on asymptotic inference.

the assignment of the mentors under the _Plus_ modality at scale. Each bar represents the treatment effect of the policy ( _α_ 1 in equation (6)) for a different sample of schools in the state of Chiapas. For the sample of schools that did not participate in the second experiment, we find that the program increases the fraction of children who enroll in secondary schools by 4.5 percentage points ( _p_ -value = 0.039), which represents an increase of 6.5 percent with respect to the sample mean. For the schools that were part of the experiment, the impact of receiving the program during the government implementation is larger (+8.5p.p., _p_ -value = 0.031), although the two estimates are statistically similar. These effects on secondary school enrollment are in line with the experimental findings on the enrollment in seventh grade (+12.4 percentage points, see Table 2). We interpret this result as evidence that the program at scale is effective in increasing schooling opportunities despite the change created by the policy implementation. Finally, the pooled estimates of the entire population of schools in Chiapas (+5.5 percentage points, _p_ -value = 0.004) are in line with the estimated effects of each subpopulation of schools.

The results for child literacy are shown in Panel B of Figure 4. After two years of rollout of the program, we find that villages that received mentors under the _Plus_ modality at scale display a 2.3 percentage points ( _p_ -value = 0.026) increase in child literacy rates when compared to villages without mentors. The magnitude of this effect implies a reduction of illiteracy rates by 20 percent with respect to the sample average. The estimated program effect for the 37
Figure 5: Policy Impact on School Closures <!-- Start of picture text --> Non−Experimental Schools Experimental Schools All Schools<br>0<br>−.05<br>−.1<br>Treat Effect on School Closure<br>−.15<br><!-- End of picture text --> _Notes_ : The bars in the figure represents the OLS estimates of the assignment to the API program during the government implementation of the _Plus_ modality on the rate of school closures as measured over the subsequent two years. Vertical lines overlaid on each bar display the 95 percent and 90 percent confidence intervals, respectively. Confidence intervals are based on asymptotic inference.

subsample of experimental schools is quantitatively similar, although a bit noisier (+3.0 percentage points, _p_ -value = 0.122). The pooled result for all schools (+2.5 percentage points, _p_ -value = 0.006) mirrors the analysis for the two subpopulations. Overall, our results confirm the conclusion that the program is scalable and it enhanced an achievement outcome for children in these disadvantaged communities.

We conclude the analysis of the impact of the mentoring program at scale by looking at school closures. As previously described, school closures represent a major threat to the scalability of the program as schools are the means through which the mentors reach the children and their families. At the same time, we also show that parents act as a channel of scalability by preventing schools from closing through increased parental engagement. To gauge whether this mechanism persists during the government implementation of the _Plus_ modality, it is important to test whether the impact of the policy on school closures is consistent across different situations for the experimental sample, as well as across different subpopulations within the state of Chiapas.

Figure 5 shows the results. The government implementation of the _Plus_ modality induces a significant and sizable effect on school closures across all the subpopulations considered.

38
When we focus on the set of schools outside of the experimental sample, we find that the program reduces the probability of a school closing by 7.3 percentage points ( _p_ -value _<_ 0.001). Schools that were part of the experimental sample also experience a sizable decrease in school closures during the policy implementation at scale, with an average impact of the mentoring program of _−_ 8.3 percentage points ( _p_ -value = 0.004). The magnitude of this effect is remarkably similar to the corresponding impact of the API _Plus_ program on school closures after two years under the experimental situation _for the same set of schools_ ( _−_ 8.3 percentage points., see Table 8). The pooled effect in the overall population of schools in Chiapas is _−_ 7.3 p.p. ( _p_ -value _<_ 0.001). This last piece of evidence strongly suggests that the underlying mechanism through which schools are more likely to remain open with the API program persists across different situations. By enhancing parental engagement, the policy implementation of the _Plus_ modality prevented disruptions in the school environment, which is a necessary condition for the success of the program at scale.

# **7 Discussion and Conclusion**

We study a school mentoring program with a home visit component in the state of Chiapas, Mexico. By exploiting two independently run field experiments, we show that relatively small differences in program design (the training module in our case) can spur substantial differences in final outcomes. We confirm that the program as it was originally implemented by the government is largely ineffective. One alternative modality of the program ( _Plus_ ) that features enhanced training for the mentors/home visitors is successful in enhancing test scores and educational attainment for the students in our sample. Parents not only increased their interactions and investment with children—a shared result among past successful interventions—but also they intensified their engagement at the school and community level.

The enhanced program modality is found to be effective after the government scale-up. The national rollout of the mentoring program, which fully converts the _Original_ modality into the _Plus_ modality with enhanced training for all the schools, provides us with an opportunity to study the mechanisms through which education interventions can be successfully scaledup. Even when the evaluation sample is broadly representative of the targeted population, changes in the implementation protocol during the transition between the field experiment and the policy rollout can threaten the success of the program at scale. In our context, school 39
closures represent a major concern during this transition. We document that the exposure to the mentoring program at scale practically eliminates this issue. Parental responses are shown to be the key mechanism through which schools remain open, thereby ensuring the viability of the mentoring program as implemented by the government. The magnitudes of the estimated impacts are remarkably comparable across situations (field experiment versus government implementation) for our experimental sample as well as for the rest of the schools in Chiapas that experienced a change in program modality (from _Original_ to _Plus_ ) during the government rollout.

Beyond the specific context of the analysis, we believe our case study can provide broader lessons for scholars who are interested in designing and evaluating scalable interventions. Whenever possible, we reiterate the importance of evaluating programs “at scale.” This represents both an opportunity to exploit the existing infrastructure of the program as well as a restriction in terms of program design, since researchers have to consider various institutional constraints and supply-side issues. While we do not necessarily advocate sampling the entire population of beneficiaries, the composition of the evaluation sample needs to reflect the targeting criteria of the intervention, and the units of analysis should be large enough so as to encompass local spillover/general equilibrium effects that likely arise in those situations. It is also crucial to consider the joint impact of the intervention on the targeted actors. In the context of mentoring and educational programs, parental responses need to be taken into account jointly with children’s outcomes both ex ante (e.g., for power calculations) and ex post (e.g., when adjusting inference procedures for multiple hypothesis testing and when conducting omnibus statistical significance tests). This is key for scalability since we have shown that it is precisely the interplay in the behavioral responses between these actors that determines the success of the programs at scale.

Finally, our work stresses the importance for scalability of the local parental incentives necessary to achieve a “Collaborative Equilibrium” in the community, as discussed in Section 5.1. Our model highlights a key trade-off between the degree of complementarity among parents, their incentive to cooperate, and the minimal actionable impact on parents that favors scaling. This result sheds light on the current debate on how to design mentoring interventions aimed at promoting better educational opportunities for children in disadvantaged contexts, including poor neighborhoods in the United States. While every parent is certainly unique to her own child (and hence not scalable), engaged communities of parents are potentially available at scale to promote the success of educational programs.

40