metadata: authors: 0: author: Outes-Leon,Ingo
1: author: Sanchez,Alan
2: author: Vakis,Renos
count: Peru
docna: 0: docna: The Power of Believing You Can Get Smarter :
            The Impact of a Growth-Mindset Intervention on Academic
            Achievement in Peru
docty: Policy Research Working Paper
subtopic: Effective Schools and Teachers,Educational Institutions & Facilities,Educational Sciences,Gender and Development
teratopic: Education,Gender
abstracts: cdata!: This paper evaluates the academic impact
            of a growth-mindset intervention on students starting the
            secondary level in public schools in urban Peru. ¡Expande tu
            Mente! is a 90-minute school session aimed at instilling the
            notion that a person's own intelligence is malleable.
            Students in schools randomly assigned to treatment showed a
            small improvement in math test scores and educational
            expectations, with a large and sustained impact in test
            scores among students outside the capital city. At a cost of
            $0.20 per pupil, ¡Expande tu Mente! was highly
            cost-effective. The results show the potential that brief
            growth-mindset interventions have for developing countries.
geo_reg_key: 119228,517191
repnme: repnme: The Power of Believing You Can Get Smarter
            : The Impact of a Growth-Mindset Intervention on Academic
            Achievement in Peru
display_title: The Power of Believing You Can Get
            Smarter : The Impact of a Growth-Mindset Intervention on
            Academic Achievement in Peru
listing_relative_url: /research/2020/02/31750899/power-believing-can-smarter-impact-growth-mindset-intervention-academic-achievement-peru
topicv3: Teachers Management m1326119 1523,Education, Skills Development and Labor
            Market m1326135 1484,Education m1327048 470,Gender m1327621 615,Psychiatry, Psychology m1504686 3613
publishtoextweb_dt: 2020-02-03T00:00:00Z
historic_topic: Education,Gender
origu: EFI-Poverty and Equity-GE (EPVGE)
no_of_pages: 38
author_keys: 0: author_key: 167914
model_extractions: Policy Research Working Paper 9141

# The Power of Believing You Can Get Smarter

The Impact of a Growth-Mindset Intervention on Academic Achievement in Peru _Ingo Outes-León Alan Sánchez Renos Vakis_ Poverty and Equity Global Practice February 2020
Policy Research Working Paper 9141

# **Abstract**

This paper evaluates the academic impact of a growth-mindset intervention on students starting the secondary level in public schools in urban Peru. ¡Expande tu Mente! is a 90-minute school session aimed at instilling the notion that a person ́s own intelligence is malleable. Students in schools randomly assigned to treatment showed a small improvement in math test scores and educational expectations, with a large and sustained impact in test scores among students outside the capital city. At a cost of $0.20 per pupil, ¡Expande tu Mente! was highly cost-effective. The results show the potential that brief growth-mindset interventions have for developing countries.

This paper is a product of the Poverty and Equity Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at rvakis@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **The Power of Believing You Can Get Smarter: The Impact of a Growth-Mindset Intervention on Academic Achievement in Peru**<sup>**1**</sup>

Ingo Outes-León<sup>2</sup> Alan Sánchez<sup>3</sup> Renos Vakis<sup>4</sup>

**Keywords** : education, disadvantaged students, growth mindset, metacognition, academic support, Peru

**JEL codes** : D91, I20

1 A preliminary version of this document was circulated as a working paper in Spanish (Outes, Sánchez and Vakis, 2017). This project was implemented with the support of the Ministry of Education from Peru. We thank Gonzalo Manrique for providing excellent research assistance, and Nicolás Pazos and María Gracia Rodríguez for their support during the implementation of the project. We are also grateful to David Yeager for his advice at an early stage of the project, and to Isy Faingold, Fabiola Cáceres, Andrea Cornejo, Luis Baiocchi, Efraín Rodríguez and the MINEDULab team for their continuous support during the implementation stage. 2 London School of Economics. Email: i.w.outes@gmail.com 3 Corresponding author. Email: <u>asanchez@grade.org.pe.</u> _Grupo de Análisis para el Desarrollo_ .

4 The World Bank. Email: rvakis@worldbank.org
# **1.** **<u>Introduction</u>**

In recent years, substantial evidence has emerged on the impact of both improving school inputs and creating incentives for students and teachers to improve school attendance and achievement in the context of developing countries (for a review, see Glewwe and Muralidharan, 2015). The goal of many of these interventions is to promote skill development, especially for disadvantaged groups in highly unequal societies. Comparatively, the role of socio-emotional interventions to help students succeed at school in low-and-middle-income countries remains less studied, yet this is a promising avenue given the existence of complementarities between cognitive and socio-emotional investments (Cunha and Heckman 2006, 2007).

We seek to generate evidence of a tool designed to improve the growth-mindset of adolescent students who are transitioning from primary to secondary. The intervention is based on the selftheories of intelligence (Bandura and Dweck, 1985; Dweck and Leggett 1988). Individuals tend to think of their own intelligence in two ways: some believe intelligence is something that is fixed (fixed mindset), whereas others believe intelligence is malleable and that it can be improved through effort (growth mindset). Self-theories of intelligence influence the way individuals respond to challenges: students with a growth mindset tend to be more interested in looking for situations in which they can learn and, eventually, improve their intelligence and academic achievement, whereas those with a fixed mindset are less likely to be interested in looking for situations in which they can learn (Dweck et al., 1995; Dweck, 2006). Throughout the late 1990s and early 2000s, studies from the field of psychology show that self-theories of intelligence of students can be modified through short-lived interventions and that this can have an impact on motivation and effort, and, eventually on academic achievement among students from disadvantaged groups.<sup>5</sup> See, for instance, Mueller and Dweck (1998), Aronson (1999), Aronson et al. (2002), Good et al. (2003), Blackwell et al. (2007). More recently, Yeager et al., (2019) show that a 50-minute, online, growth mindset intervention improved GPA scores by 11% of a standard deviation among low-achiever students in a random sample of schools in

> 5 In the United States, disadvantaged groups account for students from public schools from poor neighborhoods and with a predominance of Afro-American and Latino students.

2
the United States<sup>6</sup> (see also Paunesku et al, 2015);<sup>7</sup> Bettinger et al. (2018) adapted the tools from the Yeager et al. study in a small sample in rural Norway, replicating their results.<sup>8</sup> Also, Alan et al. (2019) randomly assigned fourth-grade students to 12 two-hours sessions in Istanbul, Turkey, and show this led to persistent improvements in math test scores by 20% of a standard deviation.<sup>9</sup> While encouraging, recent experimental evidence equally shows that these interventions can also be ineffective (Foliano et al., 2019; Ganimian, 2019). Furthermore, a recent meta-analysis that analyzes results from 29 studies shows mixed evidence of impacts on academic achievement (Sisk et al., 2018). To our knowledge, no study has demonstrated whether a growth mindset intervention can be successful in the context of low-and-middle-income countries.<sup>10</sup> With millions of students around the globe attending schools of insufficient quality 6 Yeager et al. (2019) randomly assigned a 50-minute (2 sessions, 25 minutes each), online intervention in a sample of low-achiever students from the ninth-grade. Students come from a national sample of 65 regular public secondary schools in the United States.

7 Mueller and Dweck (1998) show that students whose effort is praised improved their performance in math test scores by 30% compared to students whose intelligence is praised. Blackwell et al. (2007) administered 8-sessions of 25 minutes each (one per week) in which the science of the brain was explained to students, in particular how brain connections grow, and people become more intelligent when faced with challenges. As part of these sessions, students were asked to read a brief text that described the science behind the growth mindset (“You can grow your brain”). Subsequently, students were asked to think about examples in which their performance had improved through practice. Finally, they were asked to write a letter to younger students explaining the concepts learned –this was to help persuade them about the concepts learned through cognitive dissonance (Aronson, 1999). By the end of the academic year, an improvement in math test scores by 0.2 standard deviation was detected. Good et al. (2003) used a mentorship strategy (which included sessions for two months, and email exchanges with the mentor during the school year). They found positive effects on math test scores among women by around 0.2 standard deviation. Aronson et al. (2002) developed a similar intervention in which they worked with mentors that were in the last years of college to teach the growth mindset to first-year college students –and found an impact on final scores of 0.29 standard deviation. Paunesku et al. (2015) randomly assigned a 45-minute online session to students (the sample, 1,594 students in 13 schools, was larger than previous studies). During the session, students were asked to read an article about the growth mindset, and then to write letters to other students about the ideas learned. By the end of the semester, students from the intervention group were 6.4 p.p. more likely to have a satisfactory score in their main course.

8 Bettinger el al. (2018) randomly assigned a 135-minute, online intervention (3 sessions, 45 minutes each) among 345 students in 1 rural school in Norway.

9 Alan et al. (2019) randomly assigned the intervention at the school level in a sample of 16 schools in Turkey (Istanbul).

10 Ganimian (2019) randomly assigned a short-lived intervention at the school level in a sample of 202 urban, public secondary schools in Argentina (province of Salta), with a focus on 3
in the developing world, the potential of highly low-cost socio-emotional interventions stands as highly policy relevant. We aim to fill this gap.

We focus on Peru, a middle-income country that in recent years benefited from sustained economic growth along with substantial poverty reduction (monetary poverty reduced from 54% to 21% to between 2000 and 2015 according to official statistics), but where schooling achievement remains very low and highly unequal –by 2018, only 14% and 16% of students in the second grade of secondary level (equivalent to the eighth-grade) achieve a satisfactory performance in _Mathematics_ and _Reading Comprehension_ tests, respectively.<sup>11</sup> Within this context, we evaluate the impact of a growth-mindset intervention in Peru. The intervention, called ‘ _¡Expande tu Mente!_ ’, adapted existing tools (Yeager et al, 2016 and 2019; Paunesku et al., 2015) to a context in which online interventions are not feasible. The intervention, implemented with support from the Ministry of Education, consists of a 90-minute school session led by tutor teachers. To measure the impact of the sessions, 800 public schools were sampled in three regions in Peru. Two equally powered samples were created: a _metropolitan_ sample and a _regional_ sample. Within each sample, half of the institutions were randomly selected to participate in ‘ _¡Expande tu Mente!_ ’.

Packages were sent by courier and implementation monitored by phone. Tutor teachers from grades seven and eight (the first two years of the secondary level) were asked to deliver the sessions to all their students. Approximately 60% of all eligible schools delivered the sessions. We used information from the students’ census evaluation from 2015 and 2016 to test the academic impact of ‘ _¡Expande tu Mente!_ ’ sessions approximately 2 and 14 months after the intervention took place (respectively). Results after 2 months show that being assigned to the sessions improved _Mathematics_ test scores by 5% of a standard deviation (the point estimate for _Reading Comprehension_ is similar but statistically insignificant). We also uncover substantial geographical heterogeneity: the main result for _Mathematics_ is explained by an ITT impact of 13% in the _regional_ sample and virtually a zero impact in the _metropolitan_ sample. In the _regional_ sample, we also detect an ITT impact of 9% in _Reading Comprehension_ test students in the 12<sup>th</sup> grade (approximately 9,000 students). The duration of the Ganimian intervention is not reported, but it is based on the adapted 90-minute version reported in Outes et al. (2016). Interestingly, the author found no impact of the intervention on academic achievement, nor indeed on other dimensions.

> 11 <u>http://umc.minedu.gob.pe/wp-content/uploads/2019/04/presentacion-web-ECE2018-1.pdf</u>

4
scores. The results of the intervention are persistent. In the _regional_ sample, 14 months after the intervention those students from schools assigned to treatment that took the census evaluation improved their scores by 10%, 12% and 10% in _Mathematics_ , _Reading Comprehension_ , and _History, Economics and Geography_ (compared to the control group). Our results shed light on the potential of growth-mindset interventions. Given its per capita ITT cost, this puts the ‘ _¡Expande tu Mente!_ ’ package among the most highly cost-effective educational interventions in the developing world. At the same time, our results highlight that while growth-mindset interventions work, they do not work in all contexts. In _metropolitan_ areas, where schools and classrooms are larger and students have higher scores, our intervention failed to be effective.

The rest of the paper is organized as follows. Sections 2 and 3 describe the intervention and its conceptual pathways (respectively). In section 4, we describe the experimental design and the implementation of the intervention in the field. Sections 5 and 6 present the data and the empirical strategy, and sections 7 and 8 report our main results and robustness checks. Section 8 concludes.

# **2.** **<u>The intervention</u>**

The intervention, known as ‘ _¡Expande tu Mente!_ ’ in Peru (‘Grow Your Mind!’; from here onwards, GYM)) was a collaboration between the authors and the Peru Ministry of Education (MINEDU). Within MINEDU, the research team worked in coordination with _MineduLAB_ , the cost-effectiveness laboratory of the Ministry, and the Direction of Secondary Education. The main objective of the intervention was to instill in students the growth-mindset to improve their motivation, perseverance, effort at school, and, ultimately, academic performance (see the theory of change in **Section 3** ). The basic education level in Peru is composed of a pre-school level, six grades at the primary level and five grades at the secondary level. The intervention was designed for the early grades of the secondary level, specifically for those in the first and second grade of secondary (equivalent to the seventh and eighth grades, respectively), a transitional period that students often find challenging and which is the beginning of the path to dropping-out from school.

The key component of the intervention is a 90-minute school session designed to introduce students to neuroplasticity and the concept that cognitive abilities are malleable. It was 5
delivered during school hours by local teachers, specifically by the tutor in charge of each classroom.<sup>12</sup> The content of the session is based on previous work by Yeager et al. (2016, 2019), Paunesku et al. (2015) and Blackwell et al. (2007).<sup>13</sup> The session comprises three distinct segments of approximately 30 minutes. In the first segment, students were asked to individually read a text designed to introduce students to the concept of growth-mindset (the ‘GYM text’). This was an adaptation of the seminal intervention text in (Blackwell et al., 2007), which explains that the ‘brain works like a muscle’ and that with effort and practice anyone can improve their intelligence (see **Annex 1** ). In the second part of the session, the content of the text is debated by groups of 4 to 5 students (the composition of each group was freely decided by each teacher) and then by the classroom, with the teacher serving as a moderator of the debate. In the last segment of 30 minutes|, students were asked to write individually a reflective letter to a friend/relative describing what they learned.<sup>14</sup>

In a follow-up session, teachers were instructed to carry out two additional activities. First, to choose the best letters (according to their subjective assessment) and to hang all their student letters on the wall, alongside a poster provided by the program and which summarizes the key concepts of the session (see **Annex 1** ). Teachers were asked to leave the letters and the poster on the classroom walls for the remainder of the school year. They were to act as a physical reminder of the sessions and increase the long-term salience of their teachings. Second, teachers were asked to take a picture of the students alongside the letters and the poster and to share this picture by email.

From the outset, the intervention was designed to be highly replicable and scalable. All intervention materials reached schools by parcel delivery (the ‘GYM package’), and no ad-hoc 12 In schools in Peru, tutorial sessions take place weekly over the entire school year. The primary aim of these sessions is to provide a space to help students with their emotional and academic development, and to promote a strong relationship and better emotional understanding between students and teachers. While there are guidelines from MINEDU about the content of these sessions, ultimately it is up to each school and each teacher to define the topics to be covered.

13 Intervention tools were translated into Spanish and adapted to the cultural and curricular idiosyncrasies of the Peruvian context. Instructions for tutor teachers were further developed by the authors. Before the intervention, tools were validated in focus group discussions with local teachers and pupils at GRADE.

14 The importance of reflective writing has been shown in a multiplicity of psychosocial studies (see, for instance, Countryman, 1992), including growth-mindset interventions (e.g., Paunesku et al., 2015).

6
school visit or teacher training was required. The GYM package contained 6 elements: (1) a letter for the head teacher signed by the Director of Secondary of MINEDU, explaining the nature of the activity and requesting the school to participate; (2) individual letters to all tutor teachers within the school (who would be in charge of running the sessions); (3) detailed instructions about the structure of the session and activities to be carried out; (4) an annex with additional information about the concepts of growth mindset and examples about how to motivate the students; (5) copies of the GYM text for all students expected to receive the session; and (6) 1 poster for all classrooms to be taking part of the intervention.

Head teachers and tutors were expected to manage independently the implementation process within each school. The remote nature of the intervention raises questions of implementation fidelity and compliance. While aware of these concerns, the design was deliberate. We aim to avoid the pitfalls of small-scale evaluations that do not replicate well when scaled up (Muralidharan and Niehaus, 2017). Strategies to minimize incompliance include a tracking verification system by phone and the request of providing photographic evidence of completed sessions (see **Section 4** for a more detailed explanation).

# **3.** **<u>Conceptual framework</u>**

**Figure 1** exemplifies the theory of change from the intervention. Students are exposed to the concept that, with effort and practice, anyone can improve their intelligence. By appropriating this concept, we expect students with a fixed-mindset to modify their mental model towards a growth-mindset. Because of this new knowledge, students might increase their perseverance and motivation to learn, and adjust their aspirations and expectations, which eventually leads to improvements in the academic effort and learning outcomes. Furthermore, it is expected that there will be a feedback process by which improvements in learning outcomes lead to further improvements in perseverance, motivation, aspirations, and efforts exerted. The intervention is expected to influence the student, however by design the tutor teachers must first expose themselves and learn about the growth mindset. Therefore, through the process of studying the material to implement the GYM session, we expect that teachers either change and/or receive confirmation of their own priors, which could influence the way they interact with academically weak students.

7
The impact of the intervention on students’ skills can be illustrated through a production technology that links school inputs to the formation of cognitive skills and socio-emotional skills (including aspects such as motivation and perseverance) over multiple periods. Specifically, consider the skills formation model (Cunha and Heckman, 2008), enhanced to include the role of school inputs –as in a standard education production function, see Todd and C Wolpin (2007). Cognitive and socio-emotional skills in period t (represented by vector St and StN) depend on skills accumulated up to period t −1 (St−1C and St−1N ) and investments in both types of skills in period t. For simplicity, we distinguish between investments at the school j,School j,Others level (It ) and other investments (It ) for j = (C, N)). The skills formation process is depicted as follows, where μ<sup>j</sup> represent initial endowments. We conceptualize the intervention as an exogenous N,School increase in school-level investments in socio-emotional skills in period t (It ). Consider a three-period horizon (t −1, t and t + 1). In this case, both a contemporaneous (direct) impact on StN, and a subsequent (indirect) impact on St+1C and St+1N are expected, the latter occurs due to cross-productivity (impact of socio-emotional skills on cognitive skills) and realignments on other investments (e.g., more hours of study at home). Furthermore, the intervention is likely to modify the way the teacher interacts with students (e.g. changes in pedagogical strategies), which can have an additional direct impact on school-level investments, generating additional improvements in skills accumulation in periods t and t + 1.

# **4.** **<u>Empirical design</u>**

# (a) <u>Experimental Design</u>

The experiment took place in 2015. Three adjacent regions (of 24 in the country) were selected in coordination with MINEDU: Lima, Junín, and Ancash. These regions capture some of the geographic diversity of Peru –coast and highlands– while their relative accessibility to Lima city (from which parcels were sent) ensured reliable parcel delivery. The study focused on 8
public schools and in urban areas, the latter to reduce logistical complexity of parcel delivery— in the Lima region population is predominantly urban, whereas in Junín and Ancash approximately 2 out of 3 households are located in urban areas. Three types of schools were excluded from the sampling frame: (a) those located in districts in the top quintile of wealth; (b) those that operated night-shifts only; and, (c) those with fewer than 15 pupils in total. These restrictions resulted in a universe of 946 schools in the three regions. To maximize statistical power, it was decided to sample 800 from these 946 schools.

In total, 400 of the 800 schools were randomly selected to receive the ‘GYM package’. The study contains two equally powered samples: the _metropolitan_ samples with 398 schools from the Lima region, a densely populated area with over 12 million inhabitants, and the _regional_ sample which includes 402 urban and peri-urban schools in the regional towns of Ancash and Junín. To ensure balance in the experimental sample and maximize statistical power, treatment status was stratified across poverty quintiles and regional strata (Bruhn and McKenzie, 2008), making 12 strata in total.<sup>15</sup> **Table 1** provides an overview of treatment and control schools, across the pooled and individual experiments.

Treatment schools were targeted to receive the ‘GYM package’ and to implement the Grow Your Mind session during regular tutorial hours in 2015. In each institution that received the package, tutor teachers were asked to implement the session in every classroom for grades seven and eight. Control schools were not approached, and, thus, continued using the national curriculum in their tutorial sessions. The experiment was designed to assess the impact of the ‘GYM package’ using information from the students’ census evaluation ( _Evaluación Censal de Estudiantes_ , ECE), administered a few months after the intervention for those in the eighth grade, and more than a year later for those in the seventh grade (see **Section 4e** ).

15 Pre-intervention power calculations of the school-clustered randomized control trial suggested a low minimum detectable effective size (MDES) of 12% for the pooled sample, while individual experiments were powered to detect an MDES of at least 17%. To compute MDES in this level-3 cluster RCT, we assumed 2 sections per school with 25 pupils per section, an intra-class correlation at the school-level of 0.25 and within-school correlation across teachers of 0.15. In reality, pre-treatment power calculations were very conservative: not only average schools turned out to be larger, but the intra-class correlation across schools was closer to 0.15. Ex-post calculation based on the 2015 ECE national data suggests that the pooled and individual studies were powered to detect 8% and 12% MDES respectively.

9
# (b) <u>Implementation</u>

Packages were prepared in July 2015, at GRADE premises (in Lima). Each ‘GYM package’ was personalized for each school based on the number of classrooms in grades seven and eight, as reported in the 2014 Peru School Census.<sup>16</sup> Once packages were ready, a private courier company was hired to deliver them to all schools in the treated group. Each package contained an official letter from MINEDU (signed by the Director of the Secondary Level) addressing the head teacher of the school. This letter was placed on the outside of the package so that it was visible to the person who received it.

To locate schools, the courier was given the name of the institution, full address and (when available) contact phone numbers, as reported in the publicly available information system of MINEDU called ESCALE. The initial aim was to deliver all packages during August 2015. However, there were problems with some of the deliveries because the information given to the courier company was not detailed enough and/or was not accurate. For this reason, for a group of schools the courier company was given a physical map to locate the district (elaborated with Google Maps, using the geo-referenced location from the school) and the most up to date information about phone numbers (which was not necessarily information that was publicly available at that time). This information was obtained from the data division of the Ministry ( _Sistema de Información de Apoyo a la Gestión de la Institución Educativa_ , SIAGIE). Based on this strategy, the delivery of packages was completed between August and September 2015. As part of the protocol, each school that received the package signed a delivery receipt.

# (c) <u>Monitoring delivery and compliance</u>

A small tracking team, composed of two individuals with experience in telemarketing, was trained to monitor–through phone calls–all schools that had received the ‘GYM package’ according to the courier. The tracking exercise had multiple purposes. First, to make sure that the ‘GYM package’ had been received by tutor teachers in each school. Second, to ask if there were any doubts about the nature of the sessions and/or if the material they received was incomplete. Third, to monitor when sessions were going to take place and/or to ask why

> 16 If the number of sections was uneven, we rounded it to the next even number (e.g., 4 packages were sent to schools with 3 sections). We assumed each section had 30 students.

10
sessions were not going to take place. Fourth, to remind schools that tutor teachers had to share by email at least one picture per classroom (with the students standing up next to the letters prepared by them, and next to the GYM poster). For all these reasons (monitoring delivery and monitoring compliance of the sessions) the tracking team was in constant contact with both head teachers and tutor teachers from September and up to December 2015 (the end of the school year). This tracking exercise was complemented with massive email remainders.

The results of the delivery process and compliance are reported in **Table 2** . From the 400 schools assigned to treatment, 340 of them received the ‘GYM package’ (85%; 79% in the _metropolitan_ sample, 91% in the _regional_ sample). Those cases where the package did not reach its destination include very remote areas and/or cases in which the courier was not able to locate the school. In addition, there were a handful of cases in which the school did not want to receive the package. Considering all eligible schools ( **Table 2** , Panel A), 60% declared having administered at least 1 ‘GYM session’, whereas 55% provided physical evidence (i.e., a picture) of at least 1 ‘GYM session’. In our analysis, we define a school as a complier if there is evidence (i.e., pictures) that GYM sessions were administered in all classrooms at a given grade. Following that definition, the compliance rate was 45% in the eighth grade and 43% in the seventh grade. A larger compliance rate is observed in the _regional_ sample, however, this difference is driven by differences in the package delivery success rate; when looking at the compliance rates among those that received the package ( **Table 2** , panel B), the difference (in percentage points) reduces substantially. When sessions did not take place, the tracking team was asked to inquire about the reason. The most common reason was lack of time due to other activities.

# (d) <u>Cost</u>

Efforts to develop a scalable intervention resulted in a highly inexpensive tool. Typical schoollevel interventions incur substantial costs in teacher training and school visits; the ‘GYM package’ requires neither of them. As a result, the unit cost of “°¡Expande tu mente!” was $55 per treatment school or $0.20 per pupil (on average). Intervention costs were split between (a) courier services for parcel delivery (30%), (b) telephone operators for monitoring (20%) and (c) printing and packing of intervention materials (50%).

11
# **5.** **<u>Data</u>**

# (a) <u>Students’ census evaluation</u>

The experiment was designed to assess the impact of the ‘GYM package’ using results from the annual students’ census evaluation ( _Evaluación Censal de Estudiantes_ , ECE) administered by MINEDU. The strategy of asking teachers from grades seven and eight to implement the ‘GYM session’ generated two separate evaluation cohorts: the eighth-grade cohort assessed in the ECE 2015, and the seventh-grade cohort assessed in the ECE 2016. First introduced at the secondary education level in 2015, the ECE administers _Mathematics_ and _Reading Comprehension_ tests to all students enrolled in grade eight.<sup>17</sup> In 2016, the evaluation included a _History, Geography and Economics_ test. These are our key outcomes measures.

The ECE in 2015 and 2016 were administered in mid-November. All pupils were requested to sit their exams on the same dates throughout the entire Peruvian geography. Test content and marking were carried out centrally by MINEDU, while the administration of the tests was under the supervision of a bespoke team of enumerators. The ECE data set provides individual test scores for all participating students. Based on these scores, students are classified into four groups according to their level of learning: ‘Before beginning’, ‘Beginning’, ‘In process’ and ‘Satisfactory’. Only those in the ‘Satisfactory’ group have achieved a learning level consistent with their current grade. In 2015, this accounted for 9.5% in _Mathematics_ and 14.7% in _Reading Comprehension_ for all the student populations in the eighth grade.

Apart from test scores, the ECE also collects pupil background data via a self-administered questionnaire, which includes standard modules on household, parental and student socioeconomic characteristics, one question on education expectations (¿What is the highest education level you expect to achieve?) and, in the ECE 2015 only, two Likert scales to measure students’ perceptions about their ability to understand and complete different tasks related to math and reading comprehension (respectively). We test the impact of the intervention on education expectations and students’ self-perception of their own abilities in these two subjects to improve our understanding of the mechanisms.

> 17 Two other ECE studies take place annually in Peru, a census assessment of students in Year 2 and a sample assessment for Year 4s. For our study, only data from the Year 8 ECE is relevant.

12
# (b) <u>Descriptive statistics</u>

In **Table 3** we report school characteristics by geographical sample. The information corresponds to the 2014 School Census (Panel A to D) and ECE 2015 (Panel E). Differences between both samples ( _metropolitan_ versus _regional_ ) are substantial. Schools in the _metropolitan_ sample are larger –with more classrooms, and more students per classroom– more likely to have an afternoon shift, and in most cases have better physical infrastructure (access to sewage, existence of a science laboratory) than those in the _regional_ sample; they are also less likely to have bilingual schools and to be in districts where the main Peruvian social programs operate. Concerning students’ achievement in these schools, attainment is higher in the _metropolitan_ sample, but grade attainment is low in general; in _Mathematics_ and _Reading Comprehension_ , only 6% and 11% attain a satisfactory level in each subject, respectively.<sup>18</sup> Based on this information, we also report results from the balancing tests (see **Table A1** in the Web Appendix). Treatment and control schools appear to be similar in most observed dimensions. Some differences are observed (by day shift and sex), however, these differences are not substantial in magnitude. Unfortunately, no ECE was available before treatment to confirm balancing in the outcome variables, as 2015 was the first year in which ECE was administered in schools at the secondary level.

**Figure 2** presents kernel distributions for _Mathematics_ and _Reading Comprehension_ standardized test scores in 2015, separately for treatment and control schools, **Figure 3** presents analogous information for 2016, including also scores from _History, Geography, and Economics_ . A clear pattern arises whereby improvements are observed in all areas and both years in the _regional_ sample. In contrast, no such improvements are observed in the _metropolitan_ sample. This graphical analysis strongly mirrors the parametric analysis presented later in the paper.

> 18 Results correspond to the control group.

13
# **6.** **<u>Evaluation strategy</u>**

To empirically assess the magnitude of the impact of the ‘GYM package’, we apply the following Intent-to-Treat (ITT) estimation model, where i is an outcome variable for pupil i, in school j and randomization strata s; is a variable that takes the value of 1 if j was assigned to treatment (i.e., to receive the ‘GYM ii ii YY GGYYGG package’) status, 0 otherwise. Thus, the coefficient provides an estimate of the intent-to-treat treatment effect. We also include a full set of strata dummies, , to account for the study ββ design. When drawing inference, here and onwards we correct standard errors for school-level ii μμ clustering.<sup>19</sup> Under imperfect compliance, the ITT comparison underestimates the average treatment on the treated (ATT). Using our monitoring data, we classify schools according to whether all GYM sessions were implemented before the ECE— T T T is a binary variable that takes the value of 1 in that case and 0 otherwise. In Equation (2), we retrieve ATT estimates by instrumenting ii TT T T T with the original assignment status, , ii ii TT GGYYGG i L L T �T T i (2) ii ii<sup>+ μμ</sup> ii<sup>+ εε</sup> ii YY = αα+ ββ TT Given the random nature of the instrumental variable, with this specification, we can obtain the ATT for the sub-sample of complying schools. There are many alternative definitions for compliance in our study, ranging from whether schools received the parcel to whether objective evidence for session implementation is available. We focus on the most demanding definition: T T T = 1 if ‘all sections from a certain grade in school j (7<sup>th</sup> and 8<sup>th</sup> grade, depending on whether the focus in on medium-term or short-term estimates, respectively) provided ii TT photographic evidence of implementation’.

19 Not only is it appropriate, with a random assignment at the school level, but in the knowledge that errors are strongly correlated within schools, failure to account for this source of clustered correlation could invalidate our inferencing tests. With 799 schools in our analysis sample, we can be confident that cluster corrections will not suffer from finite sample biases **.** 14
# **7.** **<u>Results</u>**

# _<u>Short-term impacts</u>_

Results from the impact of the intervention after two months of implementation are reported in **Table 4** , for the full sample and the _metropolitan_ and _regional_ samples. In this and all other tables from here onwards, test scores are standardized to the mean and standard deviation of the control group. Results from ITT estimates (Equation (1)) are reported in **Panel (A)** . Point estimates suggest that the ‘GYM package’ had an ITT impact both on _Mathematics_ and _Reading Comprehension_ , however only the former is significant at standard levels. Specifically, students in schools assigned to receive the ‘GYM package’ increased their average test score in _Mathematics_ by 5% of a standard deviation (the point estimate for _Reading Comprehension_ is of a similar magnitude, 4%). Differences in the impact of the intervention between the two geographical samples are striking. Students in the _metropolitan_ sample appear to have not benefited from the intervention with point estimates close to zero. In contrast, benefits in the _regional_ sample are substantial: students from schools assigned to receive the ‘GYM package’ in this sample increased their test scores in _Mathematics_ and _Reading Comprehension_ by 13% and 9% of a standard deviation, respectively.

Under imperfect compliance, ITT estimates provide a lower bound of impact. LATE estimates confirm this. We use ‘schools that provided picture-verification for all classes in eighth-grade’ as the LATE definition of treatment, and assignment status as the instrumental variable. These results are reported in **Panel B** . In the full sample, the GYM intervention had an impact of 15% in _Mathematics_ test scores –the point estimate for _Reading Comprehension_ was 11%, also statistically insignificant. The impact observed in the full sample is driven by the impact of the ‘GYM package’ among complier schools in the _regional_ sample, with improvements in these subjects by 34% and 22% (respectively). In the Web Appendix, we replicate ITT and LATE estimates using school-level aggregated outcome measures (see **Table A2** ). This specification provides equivalent findings, but with larger standardized impacts.

In **Table 5** , we report results transforming the test scores for _Mathematics_ and _Reading Comprehension_ into binary variables according to whether students’ results are considered to be at the ‘Before beginning’ level (the lowest level), ‘Beginning’, ‘In process’ or ‘Satisfactory’ level (the highest level), following the standards of the school curriculum in Peru. For 15
_Mathematics_ (Panel A), improvements are observed across the whole distribution of grades. Specifically, we observe both reductions in the proportion of students in the lowest grade category (‘Before beginning’) and improvements in the proportion in the ‘In process’ and ‘Satisfactory’ categories. As before, results are driven by the _regional_ sample. Point estimates obtained for the ‘Satisfactory’ category (ITT coefficients of 1 p.p. in the full sample and 2.3 p.p. in the regional sample) represent improvement by 17% and 38%, respectively. Similar results are observed for _Reading Comprehension_ (Panel B) but in this case, point estimates are smaller and, in some cases, marginally insignificant.

Overall, our estimates show strong evidence of the substantial impact of the intervention in the _regional_ sample, but not in the _metropolitan_ sample. It is difficult to single out a specific reason that might drive these results. One possible explanation is that “ _¡Expande tu Mente!_ ” is more likely to succeed among students with low-achievement—as evidence from other countries has also shown. At the same time, schools in the _metropolitan_ sample are larger and better equipped. It is likely that that the GYM intervention might have been more salient in a regional context, where teachers have fewer teaching tools at their disposal. Besides, the fact that the average classroom size is lower in the _regional_ sample might have contributed to the success of the GYM sessions—everything else equal, a session that requires all students to read an article is more likely to be effectively delivered in a smaller classroom.

# _<u>Robustness checks and heterogeneous estimates</u>_

There might be concerns that remote schools in the treatment group were more likely not to receive the ‘GYM package’, thereby potentially generating sample selectivity. To deal with this, we report results excluding remote schools from the control group. We do this by excluding schools from the control group located in areas the courier was not able to reach. These results largely resemble those obtained from our main sample (see **Table A3** in the Web Appendix). Also, our core analysis results remain robust to controlling for school characteristics that are different between schools assigned to treatment and control.<sup>20</sup> 20 Though the stratified randomization strategy was designed to minimize systematic imbalances between the treated and control samples, this possibility could not be ruled out. A key balancing test would have been to assess whether school attainment between treatment and control schools differed before treatment. Unfortunately, no school assessment data were available before 2015. Instead, we carried out a balancing test using administrative schoollevel data obtained from the 2014 Annual School Census. Treatment and control schools do 16
We also test for heterogeneous results by gender and ethnicity. Students in the full sample are evenly distributed between males and females, and a minority (6%) has a language different than Spanish as the native tongue (11% in the _regional_ sample). Spanish native speakers have substantially higher test scores than non-Spanish native speakers, whereas males have higher scores in mathematics and females in reading comprehension—in all cases mirroring patterns observed at the national level. In **Table 6** , we report estimates of the impact of the intervention by gender and ethnicity ( _proxied_ by the native tongue). We found no evidence of differential impacts between males and females, and, similarly, no evidence of differential impacts between Spanish and non-Spanish native speakers at standard levels of significance. Looking at the point estimates by the native tongue, point estimates are substantially larger for Spanish native speakers in the _regional_ sample. At face value, this result might come as a surprise since other studies have found that growth mindset interventions tend to favor minorities that might be afflicted by racial stereotypes. We speculate that one reason that might explain why the impact of the intervention could be larger for Spanish native speakers is that a minimum level of reading comprehension is required to read and understand the Grow Your Mind article that is part of the intervention.

# _<u>Medium-term impacts</u>_

Previous results reflect the short-term impact of the GYM intervention, measured approximately two months after the GYM session took place. While the impact on that cohort is only observed once, our design allows us to measure the impact 14 months after the intervention, on those students who were in the seventh grade in 2015 and eighth grade in 2016. These results are reported in **Table 7** . Point estimates for the full sample are like those observed the previous year: the impact of the GYM intervention is persistent in the _regional_ sample, with ITT impacts by 10%, 12% and 10% of a standard deviation in _Mathematics_ , _Reading_ not differ in most tested characteristics. Balancing tests failed in only two covariates: whether the school segregated by gender; and, whether the school runs only morning shift or morning and afternoon shifts. The imbalanced variables could be driven by outliers as they represent very small distributions. For the robustness check we included the imbalanced school characteristics as controls, finding virtually the same results. Results are available upon request.

17
_Comprehension_ , and _History, Economics, and Geography_ , respectively. Consistent with previous findings, no impact is observed in the _metropolitan_ sample.<sup>21</sup>

# _<u>Mechanisms</u>_

The administrative data available do not allow us to fully uncover the mechanisms through which the intervention had an impact on students’ test scores, however, there is some information available on the self-administered questionnaire that students assessed by MINEDU have to answer. In **Table 8** we provide evidence of the impact of the intervention on educational expectations (in the short and medium term), as well as on students’ self-perception of their own abilities in math and reading comprehension (in the short-term). In the short-term, students in schools assigned to receive the ‘GYM package’ improved their expectation to attend university by 1.1 p.p. Consistent with previous findings, the ITT impact on expectations is virtually zero in the _metropolitan_ sample, and approximately 1.8 p.p. in the _regional_ sample. Results are very similar in the medium-term estimates, albeit marginally insignificant. Finally, our results do not suggest the intervention had an impact on students’ beliefs about their own abilities in the short-term. It is possible that the intervention might have had an impact on this domain in the medium-term, once students realized they had improved their performance on both subjects—potentially thanks to an increased academic effort. Unfortunately, this hypothesis cannot be tested because the scales used to measure these self-perceptions were discontinued from the MINEDU questionnaire.

> 21 By 2016, the research team knew the intervention had had an impact in the _regional_ sample and not in the _metropolitan_ sample. Because of this, we decided to randomly allocate half the schools from the _regional_ sample to receive the ‘GYM package’ so that they could repeat the session at approximately the same month of the year. The instruments and protocols were identical, and the same monitoring procedure was put in place. The only difference was that, in this case, the letter to the head teacher and tutor teachers gave emphasis to the idea that the ‘GYM session’ should be repeated. Our analysis shows no additional impact from being assigned to repeat the ‘GYM session’.

18
# **8.** **<u>Conclusions and further agenda</u>**

Peru is a middle-income country with high educational inequalities and where less than 2 out of 10 students from the secondary-level achieve an academic performance consistent with their school grade. We evaluated the impact of ‘ _¡Expande tu Mente!_ ’, a growth-mindset intervention, on standardized test scores among secondary-level students from public, urban schools in Peru. The study randomized the introduction of an in-class, 90-minute session delivered by school teachers. Our analysis shows that even when compliance was not perfect, the benefits of the intervention were relevant and long-lasting in the Peruvian context, especially in the regional context. At a cost of less than $0.20 per pupil, this intervention holds the promise of a highly cost-effective, scalable and replicable educational tool. At the same time, the fact that we were able to detect gains in the regional sample but not in the metropolitan sample shows that the way the tool is introduced (design of the intervention) and the context in which this occurs (i.e., school and teacher characteristics) both matter to understand potential gains. While the results obtained in this study are encouraging, more still needs to be learned about specific ways through which Grow Your Mind interventions can be successful in different contexts (e.g., in metropolitan areas and rural areas in Peru; in other developing countries), for different students (e.g., article versus video versus animated story version), and how this type of psychosocial intervention can complement existing interventions. To the best of our knowledge, our study is the first to successfully test the impact of growth-mindset psychosocial stimulation tools in a developing country. Moreover, few have done so at the scale reported in this study.

19