metadata: authors: 0: author: Asim,Salman
1: author: Gera,Ravinder Madron Casley
2: author: Harris,Donna Oretha
3: author: Dercon,Stefan
count: Malawi
docna: 0: docna: Does Effective School Leadership Improve
            Student Progression and Test Scores ? Evidence from a Field
            Experiment in Malawi
docty: Policy Research Working Paper
theme: Disaster Risk Management,Disaster Response and Recovery,Disability,Gender,Human Development and Gender,Education Facilities,Social Protection,Social Development and Protection,Access to Education,Education Financing,Urban and Rural Development,Teachers,Education Governance, School-Based Management,Education
lndinstr: 0: lndinstr: Investment Project Financing
lndinstr_key: 1353855
abstracts: cdata!: Evidence from high-income countries
            suggests that the quality of school leadership has
            measurable impacts on teacher behaviors and student learning
            achievement. However, there is a lack of rigorous evidence
            in low-income contexts, particularly in Sub-Saharan Africa.
            This study tests the impact on student progression and test
            scores of a two-year, multi-phase intervention to strengthen
            leadership skills for head teachers, deputy head teachers,
            and sub-district education officials. The intervention
            consists of two phases of classroom training along with
            follow-up visits, implemented over two years. It focuses on
            skills related to making more efficient use of resources;
            motivating and incentivizing teachers to improve
            performance; and curating a culture in which students and
            teachers are all motivated to strengthen learning. A
            randomized controlled trial was conducted in 1,198 schools
            in all districts of Malawi, providing evidence of the impact
            of the intervention at scale. The findings show that the
            intervention improved student test scores by 0.1 standard
            deviations, equivalent to around eight weeks of additional
            learning, as well as improving progression rates. The
            outcomes were achieved primarily as a result of improvements
            in the provision of remedial classes.
display_title: Does Effective School Leadership
            Improve Student Progression and Test Scores ? Evidence from
            a Field Experiment in Malawi
listing_relative_url: /research/2024/07/34362950/effective-school-leadership-improve-student-progression-test-scores-evidence-field-experiment-malawi
EnvironmentalCategory: B
topicv3: Education Policy and Planning m1326781 1513,Education m1327048 470,Education Quality Assurance and
            Accreditation m1583314 4483
publishtoextweb_dt: 2024-07-10T00:00:00Z
dois: 10.1596/1813-9450-10846
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
no_of_pages: 50
author_keys: 0: author_key: 1013813
1: author_key: 1343690
2: author_key: 1686712
3: author_key: 166258
model_extractions: Policy Research Working Paper 10846 Does Effective School Leadership Improve Student Progression and Test Scores? Evidence from a Field Experiment in Malawi _Salman Asim Ravinder Casley Gera Donna Harris Stefan Dercon_ Education Global Practice July 2024
Policy Research Working Paper 10846

# **Abstract**

Evidence from high-income countries suggests that the quality of school leadership has measurable impacts on teacher behaviors and student learning achievement. However, there is a lack of rigorous evidence in low-income contexts, particularly in Sub-Saharan Africa. This study tests the impact on student progression and test scores of a two-year, multi-phase intervention to strengthen leadership skills for head teachers, deputy head teachers, and sub-district education officials. The intervention consists of two phases of classroom training along with follow-up visits, implemented over two years. It focuses on skills related to making more efficient use of resources; motivating and incentivizing teachers to improve performance; and curating a culture in which students and teachers are all motivated to strengthen learning. A randomized controlled trial was conducted in 1,198 schools in all districts of Malawi, providing evidence of the impact of the intervention at scale. The findings show that the intervention improved student test scores by 0.1 standard deviations, equivalent to around eight weeks of additional learning, as well as improving progression rates. The outcomes were achieved primarily as a result of improvements in the provision of remedial classes.

This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at sasim@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Does Effective School Leadership Improve Student Progression and Test Scores? Evidence from a Field Experiment in Malawi

Salman Asim,<sup>*</sup> Ravinder Casley Gera,<sup>†</sup> Donna Harris<sup>‡</sup> and Stefan Dercon<sup>§</sup>

## _JEL Classification_ : I21; I28; C93.

_Key words:_ Education Quality; Primary School; Education Policy; Field Experiment _Registration:_ Pre-registered with World Bank’s Strategic Impact Evaluation Fund (SIEF) in May 2019. Awarded the World Bank Human Development Award for Excellence in using Evidence to build Human Capital in June 2019.

_Timeline:_ Baseline data collection was conducted prior to the beginning of implementation in April 2019. The intervention was conducted over two school years, 2018/2019 and 2019/2020. Endline data collection was conducted from October 2021 to February 2022.

> *Senior Economist, Education Global Practice, World Bank: sasim@worldbank.org. †Education Specialist, Education Global Practice, World Bank

‡Research Fellow, Department of Economics, University of Oxford §Professor of Economic Policy, Blavatnik School of Government, University of Oxford
# 1 Introduction

Evidence from high- and middle-income countries suggests that the quality of school leadership has measurable impacts on teacher behaviors and on student learning achievement (Leithwood and Sun, 2012; Branch et al., 2012; Bloom et al., 2015; Fryer Jr, 2017; Leaver et al., 2019). In a meta-analysis of six rigorous studies based on data from the United States, Grissom et al. (2021) find the impact of improvements in principals’ practices on student achievement to be nearly as large as those of a comparable improvement in teacher’s practices identified in similar studies.<sup>1</sup> The authors identify three key areas through which school leaders achieve these impacts: organizing resources strategically around people and instruction; overseeing and providing feedback on teaching; and managing and effectively communicating with teachers, parents and the community. Adopting these three broad areas as a framework, we note that the literature finds that school leaders can improve the performance of teachers by (i) motivating them to attend school regularly and fulfil their assigned duties (Leithwood and Sun, 2012; Leaver et al., 2019), and (ii) directly supporting improvement in pedagogical practices, for example by observing lessons and providing feedback (Grissom et al., 2021; Hallinger, 2005; Ingersoll, Dougherty, and Sirinides, Ingersoll et al.; Grissom et al., 2013). Finally, school leaders can inculcate school cultures that are conducive to learning and support strong identification and communication between teachers, students and parents (Akerlof and Kranton, 2002; for Economic Co-operation and Development, 2005). However, there is limited evidence on the impacts of school leadership quality on educational outcomes in low-income countries.<sup>2</sup> The apparent importance of school leadership has led to a growing body of experimental evidence on interventions to improve school leadership skills, with mixed findings. Fryer Jr (2017), de Hoyos et al. (2020) and Tavares (2015), reporting on interventions in the United States, Argentina and Brazil respectively, find significant impacts from such interventions on student outcomes, although impacts were frequently heterogenous and took 1-2 years to emerge. Studies from Mexico (Romero et al., 2022) and India (Muralidharan and Singh,

> 1 The authors identify an impact of 0.13 standard deviations (S.D.) in mathematics and 0.09 in reading from a 1 S.D. improvement in principal effectiveness, as compared to a 0.17 S.D. gain in mathematics and 0.13 S.D. in reading from a 1 S.D. improvement in teacher effectiveness.

> 2 There are reasonable grounds to expect a relatively high marginal return from improved leadership at the school level in these countries. In particular, constraints of inputs unsurprisingly tend to be higher in low-income countries, raising the importance of efficient allocation and utilization of these resources; however, misallocation and poor utilization tend to be more common in low-income countries, particularly those in Sub-Saharan Africa (Bashir et al., 2018), suggesting a greater potential gain from efforts to build the skills of principals in resource management and utilization. For example, in Malawi, the context of this study, shortages of teachers are exacerbated by widespread misallocation of teachers between grades (see section 2.3). Similarly, challenges of teacher behavior, including absenteeism and inappropriate behavior towards children, are on average more severe in low-income countries and particularly Sub-Saharan Africa (ibid.); this potentially increases the benefits to learning and progression improvements in the skills and practices of school leaders in these areas.

1
2020) observe null effects on student outcomes. Reviewing studies from 20 predominantly middle-income countries, Anand et al. (2023) finds an average impact on learning of 0.04 S.D.. However, there is a lack of experimental evidence on the impact of interventions to improve school leadership skills in low-income countries, particularly in Sub-Saharan Africa. Three studies included in Anand et al. (2023) are conducted in low-income countries. However, two of these do not report impacts on student learning or progression (Devries et al., 2015; Lassibille, 2016). The study which does report learning impacts, Blimpo et al. (2015), finds null effects .

There are reasonable grounds to expect such training to be effective in low-income contexts. In particular, while high-income countries typically have formalized merit-based promotions for head teachers, and middle-income countries including Colombia, Peru, Ecuador and Mexico have established similar systems in recent years (Commission for Quality Education for All, 2016), in most low-income countries, including Malawi, promotions are made on a discretionary basis and reflect seniority more than merit ; furthermore, the overwhelming majority of head teachers and deputy head teachers receive no training on entering their roles (Mulkeen, 2010). This suggests that there are substantial potential benefits for a well designed and implemented program of leadership training to raise the skills of head teachers, and in so doing, improve school outcomes. There is therefore a strong need for rigorous experimental evidence on the impact of such interventions in low-income settings.

This study aims to fill this lacuna in the literature by testing the impact on student progression and test scores of a two-year, multi-phase intervention in Malawi to strengthen leadership skills for head teachers, deputy head teachers, and sub-district education officials (known as Primary Education Advisers, PEAs). We implement an intervention specifically tailored to the school environment in Malawi. The intervention consists of two phases of classroom training along with follow-up visits, implemented over two years. It focuses on skills relating to school organization in a resource-constrained environment, particularly: making more efficient use of resources available at the school; motivating and incentivizing teachers to improve performance; and curating inclusive school cultures which meet the needs of all students including overage students, students with special needs, low-performing students, and girls. We conduct a randomized controlled trial, with 1,198 schools in the treatment group and a further 1,198 in the control group. The interventions are designed by a partnership of the Government of Malawi, University of Oxford, University of Malawi, and an international consulting organization, along with the World Bank; and implemented by the government and University of Malawi with support from the consulting organization.

For data, we employ a large-scale, nationally representative, independent survey of conditions, practices and learning outcomes in schools, administered by an independent research 2
firm. The survey and test instruments were designed by the World Bank’s research team in partnership with the government and country level psychometricians and research consultants. Our endline data collection took place two years after the completion of the first (main) phase of the intervention, enabling us to observe medium-term impacts on test scores.

Our study makes two distinct contributions. This study is the only the second of which we are aware to generate rigorous experimental evidence in a low-income setting on the impact of interventions to improve school leadership on learning and progression. If the interventions can positively influence student progression and test scores in Malawi, it will have important policy implications for governments in low-income countries. Second, we evaluate a government program at national scale, with leaders from 1,198 schools (more than one-fifth of all public primary schools) from all districts in Malawi participating, in contrast to pilot interventions generally evaluated at a small scale and in particular districts or regions<sup>3</sup> . This implies the results of the study are relevant to all public primary schools in Malawi and the evaluated approach, if successful, is likely to have similar impact on student achievement if scaled up.

# 2 Design

## 2.1 Research Question and General Hypothesis

Our research question is: Can student progression and students’ test scores improve as a result of a custom-designed multi-phase school leadership training program?

Our general hypothesis is: The intervention is expected to improve utilization of resources and the behavior of teachers, and create school cultures which meet the needs of over-age students, students with special needs, low-performing students, and girls; and as a result improve student progression and test scores (compared to control).

## 2.2 Theoretical Framework

Poor outcomes in education in low-income countries reflect systemic resource limitations, poor utilization of resources, and weaknesses in education production (Hanushek, 1987; Delprato and Antequera, 2021). We employ the term _education production function_ (Bowles, 1970;

> 3 For example, the previous experimental study which reports learning impacts, Blimpo et al. (2015), employs a total of 273 schools across all treatment and control groups.

3
Hanushek, 2020) to describe the capacity of schools to convert inputs – such as teachers and classrooms – into the output of learning. In low-income countries, there exists a low level of inputs in comparison to higher-income countries, as a result of limitations of the quantity and quality of available resources such as teachers, classrooms, and textbooks. In the context of low-income countries, there may be limited scope to raise the level of inputs, as a result of overall resource constraints. In Malawi, for example, it is estimated that an additional 75,000 classrooms are required at primary level alone to bring all schools to an acceptable ratio of 60 students per permanent classroom (MoEST, 2020); even adopting a low-cost model of construction, it is estimated that this would cost US$562 million, more than one year’s entire public education budget (Government of Malawi, 2022). With such large barriers to achieving significant increases in inputs, there is a need for concerted efforts to increase the education production function to achieve improvements in outcomes.

In addition, we employ the term _education production frontier_ to describe the level of education production a school can produce, assuming efficient utilization of the available inputs (Jimenez and Gonzalez, 2011). Many schools operate within their education production frontier as a result of misallocation or other inefficiencies in the utilization of available inputs. School leadership has the potential to move a school’s education production function to the frontier by maximizing the efficiency of allocation of resources. Furthermore, even where schools operate on the education production frontier, there may be potential to extend the frontier by increasing the productivity with which the school converts efficiently allocated resources into learning. School leadership has the potential to move a school’s education production frontier outward by improving teaching practices and school cultures (Figure 1).

4
<!-- Start of picture text --> Schoolswith increased<br>productivity through<br>improved teaching practices<br>" and school cultures<br>a<br>o<br>. i ——<br>a) : : a= gs Extended<br>2 : —_ education<br>g : or : production<br>=a : gen : frontier<br>~- :<br>c :<br>5 a: a<br>aASs 2” : : Schoolswith increased<br>o” :: :: resourceefficiencyallocationthrough improved<br>¢ : :<br>oY Improved : :<br>, teaching : : Current<br>y practices : .~_ === +——_ education<br>cultures =“ : frontier<br>// and school : _—er 4 production<br>i ont- ::<br>t or: :<br>f o* :<br>! - :<br>i o” F<br>! oe” Improved 5 : Existing low<br>i ”¢ utilizationresource : : : : efficiency schools<br>‘<br>,<br>if: :<br>/ _ Classrooms/Teachers<br>Differentlevels of inputs<br><!-- End of picture text -->
sources is substantially constrained in low-income contexts, and addressal of these constraints is a necessary (but not sufficient) step to improving learning outcomes (Mbiti et al., 2019; Das et al., 2013). Inefficient allocation and utilization of available resources can exacerbate these constraints. In particular, inequitable allocation of teachers to classes can exacerbate schoollevel staff shortages by leading to larger class sizes in relatively understaffed classes, with negative impacts for learning (Angrist and Lavy, 1999; Grissom et al., 2017), and weaknesses in utilization of textbooks can exacerbate limited availability (Milligan et al., 2017). School leaders can improve allocation and utilization of resources by allocating teachers equitably to classes; encouraging or mandating proper usage of textbooks; and otherwise monitoring and directing full use of available resources in school (Grissom et al., 2021).

**Teacher behavior.** Teacher behaviors are a key determinant of learning outcomes (Chetty et al., 2014). In particular, it can negatively affect students’ learning outcomes if teachers do not attend school regularly, or when in school are not in the classroom. By contrast, teachers can improve learning outcomes by employing effective pedagogical techniques including assigning, collecting and marking homework and using teaching aids such as posters and models (Eren and Henderson, 2004; Asokhia, 2009). School leaders can improve teachers’ motivation to attend school regularly and apply effort to achieve students’ learning. Although some sources of low teacher motivation, such as poor pay or conditions, may be out of the locus of control of school leaders, school leaders may be able to exercise some influence on teachers’ motivation through simple low-cost measures such as verbally praising effective or high-effort teachers. Second, school leaders can improve teachers’ pedagogical skills by observing lessons, providing feedback, and encouraging and mandating specific pedagogical techniques that are underutilized (Grissom et al., 2021).

**School cultures.** School culture may be defined as the guiding beliefs and expectations evident in the way a school operates (Fullan, 2007), including the disciplinary climate, how well students and teachers get along, and how strongly students identify with their school (for Economic Co-operation and Development, 2005). Employing evidence from the OECD’s Programme for International Student Assessment (PISA) examinations in 2000, for Economic Co-operation and Development (2005) find that an average of 8 percent of variation in PISA outcomes between schools is explained by school climates, a greater amount than school policies or resource availability. Akerlof and Kranton (2002), conducting a systematic review of case studies and other non-economic literature on successful schools, conclude that issues of culture and identity, particularly students’ sense of identification and alignment with the perceived values of a school, explain a significant portion of the success of certain schools in the face of challenges of resource availability or difficult social contexts. They note that head teachers in schools with successful cultures typically invest a non-negligible share of school resources into the creation of community and minimization of differences between students.

6
The three channels of impact of school leadership outlined above – resource utilization, teacher behavior, and school cultures – are intrinsically connected. For example, school leaders’ ability to control the allocation of resources may extend to teachers, the largest single area of expenditure in most schools – the selection and management of whom is a key driver of school culture.

# 2.3 Context

Malawi has made impressive achievements in access to schooling since the introduction of free primary education in 1994, but its education system has failed to keep pace with increased enrolments, resulting in overcrowded classrooms and understaffed schools.<sup>4</sup> These extremely poor conditions lead to high rates of grade repetition<sup>5</sup> and dropout. In 2017/18, at the time of the baseline for this study, repetition rates nationwide averaged 32 percent in Grade 1 and 24 percent in all grades.<sup>6</sup> (Ministry of Education, Science and Technology of Malawi (MoEST), 2018) Only 59 percent of students enrolling in Grade 1 remain in school to Grade 5 (ibid.). Only one in three students who enter primary education complete all eight grades (Ravishankar et al., 2016).

Even students who remain in primary school experience poor learning outcomes. At Grade 2 level, fewer than 25 percent of students achieve minimum proficiency levels in the Early Grade Reading Assessment (EGRA) conducted by the United Stages Agency for International Development (USAID). At Grade 6, fewer than 25 percent of students achieve minimum proficiency levels in the Southern African Consortium for Monitoring Educational Quality (SACMEQ) assessment in mathematics, placing Malawi near the bottom in the region. Learning outcomes also vary widely between students, even within a single class: in 86 percent of schools, according to the baseline for this study (MLSS, 2018) the gap in knowledge scores was at least one S.D., 115 points. This is equivalent to more than two years of schooling, effectively creating multi-grade classrooms. Scores are lower for girls, over-age students, and students from minority language groups (ibid.).<sup>7</sup> Malawi’s challenges are not unique: analyses of school systems in Sub-Saharan Africa have identified a set of challenges common to countries that have rapidly expanded their primary

> 4 The average class in Grade 1 has about 150 students; and in Grade 2, about 125 students (MLSS, 2018).

> 5 Most Malawian schools require students to repeat grades if they fail to pass an end-of-year examination.

> 6 Grades are known as standards in Malawi, but will be referenced as grades throughout this paper. Grades range from Grade 1-12. Primary Education is from Grade 1-8, and Secondary Education from Grade 9-12.

> 7 Percentage scores in mathematics, Chichewa, and English adjusted using Item Response Theory and converted to knowledge score mean-centered at 500 points with an S.D. of 100 points. See section 2.7 for details.

7
enrollments, including high rates of repetition and dropout in lower grades (Majgaard and Mingat, 2012; Bashir et al., 2018). However, Malawi appears to be facing greater challenges than most comparable countries. According to a recent World Bank report, among 12 sampled countries in Sub-Saharan Africa, Malawi’s students are the most likely to repeat early grades, and the least likely to remain enrolled to Grade 8 (Bashir et al., 2018), and learning levels are low even by regional standards as described above. We hypothesize that, in addition to system-wide shortages of inputs, these particularly poor outcomes also reflect (i) inefficiencies in the utilization of resources in schools, (ii) weaknesses in teaching behaviors, and (iii) school cultures which are not conducive to learning for all students (Asim and Casley Gera, 2024).

We then hypothesize that these problems reflect weaknesses in school leadership. The majority of newly appointed head teachers receive no training in school leadership when appointed and consider the position as a continuation of regular teaching duties; an official program of training, which some head teachers have attended in the past, is outdated and overly focused on administrative responsibilities (World Bank, 2015).

We further hypothesize that improvements in the skills of school leaders, engendered through a targeted and high-quality program of up-skilling, could enable improvements in practices in each of these areas and, as a result, in outcomes.

**Utilization of resources:** Malawi’s teachers and classrooms are disproportionately targeted to upper grades, reflecting poor allocation decisions by school leaders. This contributes to extremely large class sizes in lower grades: the median school has an average class size of 139 in Grade 1, but just 59 in Grade 7 (Asim and Casley Gera, 2024). Although teachers are allocated to schools by Government, their allocation to specific grades is the responsibility of head teachers, who may need to overcome pressure to over-allocate teachers to upper grades, both from teachers themselves, because of a perceived lack of prestige or promotion opportunity from working in lower grades; and from parents of students in upper grades, who prefer small class sizes to maximize the potential scores of upper grade students in the Primary School Leaving Certificate Examination (PSLCE), a high-stakes examination completed in Grade 8 which determines transition to secondary school.

Lower grades’ classrooms are also typically in worse condition than those of upper grades, with fewer chairs, textbooks and other classroom materials (ibid.). Management of materials is subject to other inefficiencies, for example textbooks being stored centrally, using lesson time for distribution and preventing students from using them for home study. Although the 8
overall supply of these inputs in schools is largely outside of the control of school leaders,<sup>8</sup> head teachers can set and enforce policies on allocation and utilization, for example by allocating classrooms and teaching and learning materials to grades and by setting and enforcing policies ensuring the use of materials in the classroom and the right of students to take them home.

Problems of resource utilization extend to the most expensive and important education input, teachers. Malawi’s teachers demonstrate extremely low levels of motivation, attendance, and time on task. At baseline (MLSS, 2018), 13 percent of teachers at sampled schools were absent on the day of data collection, and in 50 percent of observed classrooms the teacher was out of the room when enumerators entered during scheduled lesson time. Teachers’ own estimations suggest that they spend less than 2.5 hours per day teaching. In 10 percent of observed lessons, the teacher was engaged in actual teaching activities for half of the allotted lesson time or less. Only 10 percent of teachers in observed lessons set homework for students, and only 5 percent collected or reviewed homework previously assigned to students, suggesting that thousands of teachers may be routinely giving homework that is never marked.

**Teaching practices:** Malawi’s teachers are as likely as those in neighboring countries to employ a range of positive teaching practices, including correcting students’ mistakes and providing positive reinforcement. However, Malawi’s teachers lag behind on more sophisticated pedagogical techniques: only 24 percent of observed teachers at baseline used teaching aids such as posters and models to support learning; only 22 percent used local information and materials to make lessons more relevant to students; and only 56 percent of classrooms at rural schools have instructional materials such as maps and charts displayed on the walls.

**School cultures:** Many of Malawi’s schools have not successfully developed school cultures in which all students are consistently treated with respect by teachers. Seven percent of head teachers say that teachers in their schools verbally mistreat students at least once per month, and 9 percent of students say that they have been embarrassed by a teacher in front of other students. Issues of respect extend to interactions between students: one-fifth of head teachers say that students physically harm other students at least once per month. In addition, girls achieve significantly lower learning outcomes than boys as early as Grade 4, and overage students also underperform their peers (Asim and Casley Gera, 2024), suggesting that school cultures do not equitably meet the need of all students.

> 8 Teachers and classrooms are allocated to schools by Government, although schools can hire or construct more from available financial resources. Teaching and learning materials are both supplied directly to schools in response to requests by Government and donors, and obtained through grants to schools dedicated to their purchase.

9
# 2.4 Intervention

The intervention consisted of a multi-phase program to strengthen leadership skills for head teachers, deputy head teachers, and PEAs. The intervention employed both classroom training and follow-up visits to schools to measure changes in practices and behaviors and to inform a refresher phase of training conducted a year after the main training. The intervention was part of the Malawi Education Sector Improvement Project (MESIP), a major program of education investment and reforms implemented by the government from 2016 to 2021.<sup>9</sup> The intervention, known as the School Leadership Program, aimed to improve the capacity of school leaders to conduct key activities in each of the three channels identified above: utilization of resources, managing teacher behavior, and building inclusive school cultures for all students, particularly overaged students, low-performing students, students with special needs, and girls. The skills and practices encouraged by the intervention were selected based on the literature on effective school leadership practices for improvements in learning, and through consultations with Government and school leaders on particular constraints in leadership practices in Malawi’s schools.

To address inefficiencies in allocation and utilization of resources, the training included a module dedicated to _Improving school management in a resource constrained environment_ . This was intended to provide participants with skills to, among others, equitably allocate teachers to classes and improve utilization of textbooks (Grissom et al., 2021) and classrooms. We anticipate that this module would lead to improvements in the allocation of teachers between grades, leading to improvements in pupil-teacher ratios (PTRs) in lower primary (Grades 1-4), and improved textbook utilization practices, leading to improvements in observed pupiltextbook ratios in class.

To address weaknesses in teacher behavior, including absenteeism and low time-on-task, the training included a module on _Improving teachers’ motivation and rewarding performance_ . This was intended to provide participants with skills to support and reward teachers’ pedagogical performance (Ingersoll, Dougherty, and Sirinides, Ingersoll et al.; Grissom et al., 2013); in particular to, among others, identify issues affecting teacher morale and motivation and

> 9 MESIP was financed by the Global Partnership for Education and was complemented by MESIP-Extended, a partner project financed by the Royal Norwegian Embassy which extended the MESIP interventions to additional schools. Our first cohort of schools represents schools treated under MESIP and our second, schools treated under MESIP-Extended. The implementation of the intervention was conducted concurrently for the two cohorts with the same tools, staff, and procedures.

10
provide support; assess teacher performance and identify strong performers and problem areas; reward strong performance using social awards, recognition in school assembly and other low-cost methods; and address problems through model lessons, in-service training and other methods. We anticipate that this module would contribute to improvements in the frequency with which head teachers observe teachers, provide feedback, and reward teachers for strong performance, leading to improvements in teacher attendance, effort and teaching practices.

To address problems of school culture, the training included a module on _Creating a positive and inclusive culture towards over-age children and girls_ . This was intended to provide participants with skills to, among others: understand, monitor and prevent inappropriate treatment by teachers and students of over-age children, learners with special needs and girls, and its impact on well-being and learning outcomes; train teachers to identify, encourage and support low-performing students; and monitor and address bullying. We anticipate that this module would lead to increased Head Teacher effort to prevent bullying, leading to reduced incidence of bullying.

Finally, the training included a module on _Maintaining and promoting utilization of financial and management records_ . This supported improved resource utilization, by training participants to maintain full and up-to-date records of school income and expenditures; utilization of textbooks and other non-financial inputs; and teacher attendance and time on task. It also supported improvements in teaching practices and school cultures by training participants to maintain full and up-to-date academic records, including on student attendance and learning, measured through continuous assessment; and to use these records to monitor and address poor teaching practices and chronic student absenteeism and low teacher effort.

We anticipate that, if the intervention were successful, the improvements in resource utilization, teaching practices, and school cultures engendered by the intervention would ultimately lead to improvements in student progression and dropout. See section 2.5.

The intervention was developed by a consortium of The Ministry of Education of Malawi (MoE, particularly the Department of Teacher Education and Development and Directorate of Basic Education; Asian Institute of Developmental Studies, Inc. (AIDSI), a Malaysia-based consulting organization selected by MoE through tender; The School of Education of the University of Malawi (SEUM), acting as local partner to AIDSI; the University of Oxford; and The World Bank, which provided technical support and guidance to the development of content. Following initial development of the materials and protocols, implementation was carried out by the MoE in partnership with AIDSI and SEUM.

The intervention consisted of three main phases: 11
_Main training:_ This consisted of 10 days’ intensive classroom training, conducted in batches of 100 participants (divided into smaller groups for particular activities). The training included a mix of direct instruction, focused on building awareness of appropriate practice and behavior, and interactive practical exercises, focused on developing goals for improvement. A pre-test and post-test were administered to participants at the start and end of training to measure the short-term gain in knowledge; the average score increased from 48 percent to 72 percent, suggesting gains in knowledge at least in the short term.

_Follow-up visits:_ This consisted of three follow-up visits around two to six months after initial training. The first visit was conducted by one of the consortium’s trainers, the second and third by the PEA of the zone who had also undergone the main training. The visits observed behavior change by participants and provided instruction and guidance on correct implementation of the strategies and approaches learned during classroom training, as well as informing the design of the refresher training (see below).

_Refresher training:_ This consisted of three days’ classroom training, focused on areas identified as having seen limited improvement in follow-up visits and by a third-party monitoring firm.

## 2.4.1 Impact of COVID-19 on intervention

The main training and follow-up visits were completed in April -December 2019, prior to the onset of the COVID-19 pandemic. The refresher training took place in August-September 2020 during a period when Malawi was enacting restrictions (including the closure of schools) as a result of the pandemic. Batches were divided into classes of 50 or fewer to comply with regulations on gatherings. No changes were made to the content or schedule of training.

# 2.5 Theory of Change

If the training is effective, we anticipate that the intervention will have direct impacts on the attitudes and behaviors of participating school leaders. In particular, we expect to observe increased incidence of the practices specifically targeted by the training, including: improved distribution of teachers between grades, measured through the teacher utilization ratio between lower and upper primary; more frequent observation of teachers by head teachers and more use of rewards for teacher performance, supporting improved teacher attendance, effort and teaching practices; and improved efforts by head teachers to prevent bullying.

We then anticipate intermediate impacts as changes in school leaders’ behaviors and practices 12
<!-- Start of picture text --> Increased Allocation of teachers to grades Lower primary pupil-teacher ratio<br>efficiency Pupil-textbook ratio<br>(moving to the Improved resource<br>education utilization . Teacher attendance<br>production Observation of Student progression<br>frontier) teachers/<br>- classrooms Rewards for Teacher effort<br>teacher<br>f . performance Teaching practices<br>Increased Improved teaching<br>productivitya Practices Remedial- classes<br>th (extening 4 Student test scores<br>e educa fon Teacher bullying<br>production Improved school Head teacher effort to prevent<br>frontier) cultures bullying Student bullying<br><!-- End of picture text -->
(DiD) estimator would require twice as much sample to yield the same power when baseline is taken, and autocorrelations are low (McKenzie, 2012).

For the empirical analysis, we expect two additional assumptions to hold in our ANCOVA specification. (1) Independence of the covariate and treatment effect: as schools are randomly assigned to treatment groups, we find independence between our covariates and treatment (see **Tables 1 and 2** ); (2) Homogeneity of regression slopes: we expect a positive slope between the covariate and the outcome across all treatment groups. We construct two separate specifications, the first (Equation 1), measures the student-level effects, and the second (Equation 2) measures the school-level effects.

For the hypotheses that relate to individual, pupil-level outcomes, we report treatment-effect estimates based on a simple comparison of post-treatment outcomes for treatment and control individuals.

In Equation 1, _Y_ is is the outcome of interest for student _i_ in school _s_ at endline. _β_ is the coefficient of interest, capturing the effect of the intention to treat (ITT). _T_ is an indicator for whether school _s_ was randomly assigned to Treatment, _Y_ 0 captures the outcomes of interest at baseline. _X_ 0 is a vector of baseline covariates. _ε_ is a mean zero error term. We cluster our errors at the school level, the unit of treatment assignment.

To investigate the school-level effects, we report treatment-effect estimates based on a simple comparison of post-treatment outcomes for treatment and control schools. In Equation 2 where _Y_ is the outcome of interest for school _s_ and _T_ indicates that the school was assigned to treatment, _Y_ 0 captures the outcomes of interest at baseline. _X_ 0 is a vector of baseline covariates. _ε_ is a mean zero error term.

# 2.7 Outcomes

Our primary outcomes of interest are student progression and test scores.

14
# 2.7.1 Student progression

Rates of student progression are low in Malawi, as a result of high rates of dropout and repetition, driven in part by large class sizes and limited teacher effort. We anticipate improvements in class sizes, as a result of improved allocation of teachers to lower grades. We also anticipate improvements in teacher effort. As a result of these, we anticipate improvements in student progression, particularly in lower grades.

Our primary measures of student progression are dropout and repetition rates.

**Dropout Rate:** This is measured by the number of students in each grade dropping out in the most recently completed school year, divided by the total number of students enrolled in that grade for that year.

**Repetition Rate:** This is measured by the number of students in each grade repeating a grade in the most recently completed school year, divided by the total number of students enrolled in that grade for that year.

As these measures are drawn from school records, we measure them at the school level and report them overall across grades, and for lower and upper primary (Grades 1-4 and 5-8 respectively).<sup>11</sup>

# 2.7.2 Student test scores

As a result of improvements in class sizes, teacher effort, and school learning environments as described above, we anticipate improvements in the test performance of sampled students in learning assessments in mathematics and English. We utilize data from norm referenced assessments that cover competencies from Grade 1 through Grade 6 (see section 2.8. We measure these outcomes at the student level through a longitudinal sample (see section 2.9.4). Raw scores are converted to _knowledge scores_ on a mean-centred scale using Item Response Theory (see section A).

**Mathematics Knowledge Score:** This is measured by the knowledge score of each student in the mathematics test for each school.

> 11 Owing to the fact that our data is not collected annually (see section 2.8), we are unable to use standard definitions which divide the number of repeaters by the previous years’ enrollment.

15
**English Knowledge Score:** This is measured the knowledge score of each student in the English test for each school.

**Total Knowledge Score:** This is measured by the knowledge score of each student averaged across both subjects for each school.

We also measure intermediate impacts variables, combined into Anderson indices: _resource utilization_ , including the ratio of PTRs between lower and upper primary, lower primary PTRs, and lower primary pupil-textbook ratios; _teacher management_ , including frequency of observation of teachers by head teachers and of use of rewards for teacher performance; _teacher effort_ , including teacher attendance and the share of teachers at a school who were found in the classroom on MLSS visit; assigned or marked homework during observed lessons; reported satisfaction with their job; or were above the median in terms of the amount of time they spent teaching on a typical day; _teaching practices_ , including whether teachers asked students questions, employed teaching aids, and employed local materials to help deliver lessons; _remedial classes_ , including the frequency of such classes and the share of grades in which they are offered; and _school cultures_ , including the existence of clear rules on bullying and the incidence of bullying by teachers and students. For more detail on the construction of these indexes and the underlying variables, see section A.0.2.

# 2.8 Data

We draw on data from the Malawi Longitudinal School Survey (MLSS). The MLSS is an independent, nationally representative survey that provides data on conditions, practices and outcomes in Malawi’s primary schools. The MLSS was designed to align with the MESIP project cycle to both provide information to the government on implementation and outcomes of the project and to support a number of impact evaluations of pilot interventions under the project, following rigorous experimental designs; as well as to inform general education policymaking. Visits are unannounced.<sup>12</sup> Data is collected through observations and interviews – for example, observations of school and classroom facilities; observation of lessons and teaching practices; interviews with head teachers, teachers, and members of community committees; and interviews and testing of Grade 4 students, as well as a longitudinal sample of students in later rounds.

The MLSS has its own randomly selected, nationally representative sample of 924 schools

> 12 The first visit to each school is unannounced to help capture the real situation of the school in terms of infrastructure, school performance, school and classroom management practices, student and teacher absenteeism and student learning outcomes. If required to complete all instruments, a second visit is made on an announced or pre-scheduled basis.

16
which overlaps with, but does not encompass, the treatment and control samples for this intervention. MLSS baseline data collection was conducted in 733 of these schools across multiple rounds between May 2016 and December 2018 (Asim and Casley Gera, 2024).<sup>13</sup> All these rounds were conducted prior to the intervention. 659 schools underwent endline during September 2021 - February 2022. This endline data is used for measurement of outcomes of interest. For more details of the MLSS instruments and procedures, see section A.

# 2.9 Treatment Allocation and Sampling

## 2.9.1 Defining the population frame

To select school communities for treatment and control, we begin by drawing schools from the total population of public primary schools in Malawi.<sup>14</sup> We use administrative data, the 2017 Education Management Information System (EMIS) dataset (MoEST, 2017), which includes 5,552 public primary schools within 34 education districts.<sup>15</sup> Due to its pilot nature, the government requested the intervention to be implemented in schools with both lower and upper grades at primary level. Therefore, we exclude ‘junior primary’ schools which include only Grades 1-4. This leaves 5,405 schools remaining across all 34 districts.

## 2.9.2 Randomized allocation to treatment

A total of 1,198 schools were randomly assigned to treatment and 1,198 to control, across two cohorts of schools. In the first cohort, 800 schools were randomly assigned to treatment and 800 to control. In the second cohort, 398 were assigned to treatment and 398 to control. In both cohorts, the treatment group receives the intervention and the control group receives no intervention. There are no significant differences between the treatment and control groups and population (see Appendix B), and **Table 1** demonstrates there are no significant differences in characteristics between the pooled treatment and control groups.<sup>16</sup>

> 13 Baseline could only be completed in 765 of the 924 schools owing to budgetary and logistical constraints. The authors do not find any significant differences for key school-level variables in the reduced 765 school sample using administrative school census data. Results available on request.

> 14 This population excludes private schools, but includes schools owned by religious institutions which are accredited and financed by Government and considered part of the public school system.

> 15 Malawi has 28 local government authorities, including districts and municipalities. Some of these are subdivided into two or three components for the purposes of education management, producing a total of 34 education districts. In this paper, we use ’district’ to refer to these education districts.

> 16 There are similarly no significant differences between the treatment and control groups within either cohort (see Appendix B)

17
# 2.9.3 Impact Evaluation Sample

Our impact evaluation sample is comprised of those intervention and control schools which are also part of the sample for the MLSS.

# _MLSS sample_

The intervention and MLSS are run independent of each another. The MLSS sampling was conducted prior to the randomization using the same population frame.<sup>17</sup> The MLSS sampling uses probability proportional to size (PPS) to keep the sample representative not only at a national but at a sub-regional (division) level. Malawi has six education divisions across its three regions;<sup>18</sup> our strata are defined by division, with an additional stratum for all schools within urban locations. The number of schools selected within each stratum depends on the size of that stratum (section A.0.2). The final MLSS sample consists of 924 schools across all districts of Malawi and is representative at division level (see Appendix B).<sup>19</sup>

# _Overlap between treatment/control sample and MLSS_

The scale of our intervention, targeting 1,198 schools across Malawi, enables us to exploit the overlap with the nationally representative MLSS. Of the 924 schools in the MLSS sample, 386 were among the 2396 randomly assigned to treatment or control, of which 202 are treatment schools and 184 are control schools.<sup>20</sup> These schools are used for the evaluation.<sup>21</sup> The full overlap of 386 schools was included in the endline data collection and is exploited for the ANCOVA specification.<sup>22</sup> This overlap is adequately powered to detect an impact on test scores of approximately 0.2 S.D. (see section 2.9.5). This impact evaluation sample is balanced between treatment and control on key variables ( **Table 2** ).<sup>23</sup>

> 17 Like the intervention sampling, the MLSS sampling uses a population frame of all Malawi public primary schools. However, the MLSS sampling does not exclude junior primary schools.

> 18 Divisions are sub-regional administration bodies established for management of education.

> 19 MLSS baseline data (Asim and Casley Gera, 2024) shows that variability in inputs and outcomes is substantial at sub-district level (the lowest unit) and shows limited differences between three regions and six divisions (overall).

> 20 Of these, 258 are from the first cohort, of which 133 are treatment schools and 125 are control schools; and 128 are from the second cohort, of which 69 are treatment schools and 59 are control schools.

- 21 Owing to an error in implementation planning, the initial batch of training included 51 schools which were not part of the treatment group. Eight of these additional schools are in the control group. However, none of these are in the MLSS overlap and therefore not in the IE sample.

- 22 Owing to financial constraints, 337 of these were visited for the baseline round. As we employ a longitudinal sample of students, only this subset of schools is used for analyses including student test scores. There are no significant differences between this subset of schools and the rest of the impact evaluation sample (see Appendix B).

> 23 The impact evaluation sample is also balanced within each cohort (see Appendix B).

18
# 2.9.4 Student and teacher sampling

A number of outcomes of interest, including test scores, are drawn from a sub-sample of students and teachers that are interviewed and tested as part of MLSS. For teachers, the MLSS samples 10 teachers per school at baseline, all of which are re-surveyed at endline if eligible, with additional teachers added to replace those which become ineligible for interview and testing (for example, having left teaching). See section A.0.2 for details.

For students, the study uses a longitudinal sample. At baseline, a gender-balanced random sample of 25 Grade 4 students was sampled in each school. At endline, a subset of 15 students were randomly selected for re-survey at endline.<sup>24</sup> Students who had transferred to new schools were tracked to the new schools, and students who had dropped out were tracked to their homes where possible. The longitudinal sample enables us to measure the impact of the intervention on individual students’ learning trajectories, measured through endline test scores. The median student from the longitudinal sample was in Grade 7 at endline.<sup>25</sup>

# 2.9.5 Power Calculations

We follow convention in the social sciences for power calculations, using a significance level (probability of Type I error) of 0.05 and power (probability of avoiding a Type II error) of 0.8.

We are interested in calculating the endline sample size necessary to detect an effect of _∼_ 0.2 S.D. in student learning, our main outcome of interest, in each of the two specifications.

Our randomization takes place at the school level. We use the mean score for the baseline control group of 500, S.D. of 115, cluster size of 15 students per school. The intra-cluster correlation is 0.28. The required sample size to achieve this effect is 206 (103 in the treatment group and 103 in the control group).

For the identification of the effects of the intervention, our MLSS overlap exceeds the necessary sample size for both treatment and control. We are reasonably powered for subgroup analysis.

> 24 A reserve sample of two additional students from the baseline sample was used to replace students who were ineligible for the interview and testing (for example, who had died) or were untraceable.

> 25 At endline, in addition to the tracking and re-testing of the longitudinal sample, a new gender-balanced random sample of 15 Grade 4 students was sampled. Findings for this Grade 4 sample are presented in Appendix B.

19
# 2.10 Study Timeline

The study was conducted over six years. Baseline data collection took place in two rounds between April 2016 and October 2018. Development and pre-piloting of the intervention was conducted between October 2018-March 2019, with the intervention delivered in two rounds between April 2019 and September 2020 along with midline data collection. Endline data collection was conducted in October 2021 - February 2022.

# 2.11 Compliance

Noncompliance is feasible as a result of non-attendance by participating head teachers and/or deputy head teachers in one or more aspect of the training.<sup>26</sup> For the first cohort of schools, full records of attendance are available to the impact evaluation team for review and identification. We find that 743 schools attended fully (meaning that both Head Teacher and deputy Head Teacher attended both the main and refresher training). Employing a looser standard, we find that 798 of 800 schools in the treatment group participated at least substantially (meaning that at least one of the Head Teacher and deputy Head Teacher attended both the main and refresher training). In the second cohort, 385 of 398 schools in the treatment group attended at least partially. These high rates of attendance reflect a number of factors in the design and implementation of the intervention, including the context of a large-scale investment project, MESIP, which ensured a significant level of inputs in terms of staffing and management; provision of allowances for travel, accommodation and food, ensuring no cost to participants; and intensive effort by the implementing team to ensure attendance. Balance checks confirm there are no significant differences between the participating group and the entire sample for either the treatment/control or IE samples, in either cohort (see Appendix B).

# 2.12 Attrition

The potential sources of attrition at endline are students transferring to other schools, or dropping out; and teachers transferring to other schools, retiring, dying, or leaving teaching. To minimize attrition, students and teachers who have transferred are tracked to their new school to complete interview and learning assessment; students who have dropped out are tracked to their homes. In total, we reached 85 percent of the longitudinal sample of students, and 80 percent of teachers. There are no systematic differences in attrition for either students or teachers between treatment and control group ( **Table 3** ).

> 26 As the evaluation is conducted at the school level and PEAs operate at subdistrict level, we do not consider PEAs in the compliance analysis.

20
# 3 Results

**Tables 4 to 13** present ITT findings using ANCOVA. All student-level findings use Equation 1, while all school-level findings use Equation 2.

## 3.1 Test Scores

Student level results on test scores, are reported in **Table 4** . We include controls for students’ baseline scores. We observe significant impacts from the treatment. Students in (or tracked from) treated schools achieved scores an average 13 points higher in mathematics scores, equivalent to 0.1 S.D. English scores also appear to have been increased by the treatment, but the result is not statistically significant. A statistically significant impact of 10.4 points in total scores is observed, but we cannot reject the hypothesis that this is driven by the impact on mathematics scores. We present findings for the same sample of students, without controlling for baseline scores, in Appendix B; we observe similar impacts with a slightly stronger treatment effect.

## 3.2 Student Progression

**Table 5** presents the school-level results on dropout and repetition rates. We find evidence that the intervention significantly reduced repetition rates in lower primary by 2.2 points compared to control, approximately a nine percent reduction. Upper primary repetition rates also appear to have been reduced by the intervention but the result is not statistically significant. We do not observe significant impacts on dropout rates.

## 3.3 Heterogeneity

Malawi’s schools operate in a diverse range of conditions, with wide variation in remoteness, staffing, and learning outcomes, while Malawi’s students also possess a wide range of different characteristics with significant impacts on learning (Asim and Casley Gera, 2024). In order to investigate the differential impact of the intervention on schools in different contexts, we conduct heterogeneity analysis. Tables 6 to 9 show student-level estimates on test scores interacted with dummies for the relevant characteristics. As an additional check, we conduct sub-group analysis presenting ITT findings for particular sub-groups of the school and/or student sample (see Appendix B).

**_PTR._ Table 6** shows estimates on test scores interacted with a school-level dummy for whether a schools’ PTR in lower primary was below the median at baseline. We observe significantly different impacts from the treatment for students in these schools, with English scores in

21
treated schools 23 points above similar control schools. We conclude that the treatment was most effective in schools with smaller PTRs.<sup>27</sup>

**_Gender._ Table 7** shows estimates on test scores interacted with a student-level dummy for female students. We do not observe significantly different impacts from the treatment for female students.

**_Age._ Table 8** shows estimates on test scores interacted with a student-level dummy for students who were below the median age for the sample at baseline. We do not observe significantly different impacts from the treatment for these students.<sup>28</sup>

**_Score at baseline._ Table 9** shows estimates on test scores interacted with a student-level dummy for students whose scores were below the median at baseline. We observe increased treatment effects for these students, with mathematics scores increasing by 17 points as a result of the intervention. This suggests that the intervention supported students with lower levels of learning to catch up to their peers as they progressed through upper grades.<sup>29</sup>

# 3.4 Robustness Checks

3.4.1 Alternative explanations for learning impact.

In order to confirm that the observed impacts on test scores in the longitudinal sample are real, we test a number of potential alternative explanations.

First, we explore the possibility that the treatment reduced dropout among the longitudinal sample. As dropped-out students are expected to learn less than those still enrolled, a reduction in dropout could potentially explain the effects of the treatment on learning in the longitudinal sample. However, as noted above, we do not observe impacts from the treatment on school-wide dropout rates ( **Table 5** ).

> 27 We observe similar results in our sub-group analysis (Appendix B); the treatment impact on total scores was 22 points in schools with low PTR at baseline, versus 11 points in the entire longitudinal sample.

> 28 In the sub-group analysis (see Appendix B), we do observe lower impacts from the treatment for younger students which do not attain statistical significance, which may suggest that the impacts of the treatment were smaller for younger students in the longitudinal sample but that our heterogeneity analysis is underpowered to detect this effect.

> 29 We observe a similar effect in the sub-group analysis (Appendix B) in which stronger impacts are observed on mathematics scores (21 points) and significant impacts on English scores (15 points) for students with below median scores at baseline. This suggests that the overall effects on the intervention on the longitudinal sample were driven in large part by gains among those students whose performance was low at baseline.

22
As a further check, we test for differences in the share of students from the longitudinal sample who were found to have dropped out at endline ( **Table 3** ). We find no significant difference in this share as a result of the treatment. We then explore the possibility of selection bias: if the rate of attrition was higher in the control group than in the treatment group, or vice versa, this could lead to the observed impacts if higher-performing students were more or less likely to remain in the sample. However, we find no difference in attrition between the treatment and control groups.

### 3.4.2 Spillovers

Spillovers from the treatment to control group are feasible as a result of informal communication and collaboration between head teachers, deputy head teachers, and PEAs, and learnings by PEAs applied to other non-trained schools in their zones. To address this, we conduct subgroup analysis excluding control schools within the same zone as treated schools. We find no substantial differences from the main ITT estimates (see Appendix B).

## 3.5 Mechanisms

**Table 10** presents school-level results on our exploratory intermediate indicators, grouped into indices. We observe significant impacts on the index combining measures of the provision of remedial classes. We do not observe significant impacts on other mechanisms from the treatment.

To disaggregate the impacts on remedial classes, **Table 11** presents school-level results for the individual indicators that make up this index. We find significant impacts from the treatment on the average frequency of remedial classes, the share of grades in which they are offered, and the share of grades in which they are offered for free. We find the impacts on the share of grades in which remedial classes are offered to be focused on the upper grades.

Additional tables in Appendix B present the breakdown of the mechanism indicators for the other indices. We observe indicative significant impacts from the treatment on pupil-textbook ratios in lower primary and on the likelihood that head teachers reward teachers for strong performance.

# 4 Discussion

The results suggest that an intervention targeted to improve leadership skills in a low-income context can improve students’ test scores and progression in a short time period. We find 23
significant impacts on mathematics test scores from the intervention, and significant impacts on repetition rates in lower primary. The coefficient on mathematics scores, 13 points on our 500-centred IRT scale, is equivalent to 0.1 S.D., which compares favorably with the average impact of 0.04 S.D. observed in high- and middle-income countries in Anand et al. (2023). This impact is equivalent to approximately eight weeks’ learning.

In terms of the mechanisms of impact, the most significant observed impacts on intermediate indicators relate to remedial classes, specifically the incidence, frequency, and share of grades in which remedial classes are offered. This is commensurate with the finding from our heterogeneity analysis, that the benefits to learning accrued to those students in the longitudinal sample which had learning scores below the median at baseline, who would be the most likely to benefit from remedial classes.

Recalling our theoretical framework, we conclude that the intervention helped improve schools’ education production functions by increasing the amount of learning produced with the existing level of inputs. More specifically, the intervention helped schools move towards the education production frontier by maximizing the effort made by teachers to support lowperforming students. However, we do not observe impacts on teaching practices or school cultures, suggesting that the intervention did not enable schools to extend their education production frontier outward.

# 4.1 Cost Effectiveness

To estimate the cost effectiveness of the intervention, we consider the marginal cost of delivery of the main training package following initial design activities, including printing of materials, allowances, and staff costs; and the average number of students in upper primary in treated schools who are exposed to the potential benefits. We estimate the per-student cost of the intervention at US$5.23. This can be considered a conservative estimate as it does not reflect the cost of the follow-up visits or refresher training. As noted above, the learning impacts observed from the intervention on the longitudinal sample of students are equivalent to approximately 0.1 S.D. Using the J-Pal recommended approach to cost-effectiveness analysis (Bhula et al., 2022), this equates to a cost of US$53 per S.D. of additional learning, or 1.88 S.D. per US$100 invested. This compares favorably with other recent experimental trials in low-income countries including conditional cash transfers in Malawi; provision of textbooks and contract teachers in Kenya; and remedial education in India (ibid.).<sup>30</sup>

> 30 Using the entire cost of the intervention, rather than the marginal cost, the estimated impact falls to 0.97 S.D. per US$100, which remains competitive with a number of similar recent experimental interventions.

24
## 4.2 Sustainability

The intervention benefited from a significant amount of national and international expertise being brought to bear at the design stage, through the partnership between the government, Universities of Oxford and Malawi, and AIDSI. This public-private partnership enabled the development of concise, innovative content which was customized to the Malawi context while embodying international best practice. Crucially, while the entire partnership contributed to the design of the intervention, implementation was led by the government and SEUM with AIDSI providing support to logistics and administration. Prior to the completion of the intervention, a handover process was conducted to enable the continued implementation of the training by MoE in partnership with SEUM.

In 2022, a Memorandum of Understanding was signed between the local partners, MoE and SEUM, to enable SEUM to continue to support the implementation of the intervention. Following the completion of the evaluation of the pilot and the positive impacts observed, the intervention is now being scaled up to all school leaders who have not yet received the training, with financial support from the Global Partnership for Education and World Bank as part of the successor project to MESIP, the Malawi Education Reform Program (MERP). It is expected that the continuing participation of SEUM will help to ensure that the impacts are maintained at the scale-up stage.

# 5 Conclusion

We test the effect of training school leaders in 1,198 schools to improve use of school resources, motivate teachers and create an inclusive school culture for students to attain their maximum learning potential. We find that exposure to the intervention produced significant improvements in mathematics test scores for a longitudinal sample of students, equivalent to approximately 0.1 S.D. or eight weeks’ learning. We also find that the treatment significantly reduced repetition in lower grades.

The findings suggest that improvements in school leadership can be achieved in a low-income context, with Government implementation and at scale, through a multi-phase intervention combining classroom training with follow-up visits and refresher training; and that these improvements can ultimately lead to improvements in student test scores and progression.

25
# Acknowledgments

We thank the Ministry of Education, in particular Honorable Minister Madalitso Kambauwa Wirima, Secretary of Education Dr Mangani Katundu, and Director of Basic Education Grace Milner; former Directors of Education Planning Rodwell Mzonde and Francis Zhuwao, and former Directors of Basic Education Joseph Chimombo and Gossam Mafuta, for support in data collection; Muna Salih Meky, Safaa El-Kogali, Hugh Riddell, Greg Toulmin, Inam Ul Haq, Sajitha Bashir, Marina Bassi, and other World Bank colleagues in Malawi and elsewhere for encouragement, support and advice; Martin Moreno, Chris Burningham, Xu Wang, Miranda Lambert, Radhika Kapoor, Stefan Nippes, Gift Banda, Linda Matumbi, Leah Mziya, and Archit Singal, for invaluable research assistance; Vigdis Aaslund Cristofoli and Elin Ruud, Royal Norwegian Embassy, and Arianna Zanolini and Sabina Morley, UK Foreign, Commonwealth and Development Office, for comments and suggestions, and funding for the preparation of this paper. The findings, interpretations, and conclusions expressed herein are our own and do not necessarily represent the views of the World Bank, its Board of Directors, or any of its member countries. All remaining errors are our own.

# Administrative information

## Ethics approval

The Malawi Longitudinal Schools Survey has been subject to ethics review and clearance by the National Commission of Science and Technology for Malawi. The Principal Investigator, Salman Asim, has completed the training “Protecting Human Research Participants” provided by the National Institutes of Health (Certification Number: 921497; Date of completion: 05/16/2012)

## Funding

Principal financing for data collection was provided by the Royal Norwegian Embasssy, Malawi, and the United Kingdom Foreign, Commonwealth and Development Office (FCDO). The interventions were financed by the Global Partnership of Education through the MESIP program and the Royal Norwegian Embassy through MESIP-Extended.

# Tables

26
Table 1: Balance: Impact Evaluation Sample: Treatment vs Control

|Variable|N|(1)<br>Control<br>Mean/SE|N|(2)<br>Treatment<br>Mean/SE|T-test<br>Diference<br>(1)-(2)|
|---|---|---|---|---|---|
|Number of Toilets|1197|14_._335<br>(0_._284)|1194|14_._091<br>(0_._262)|0_._244|
|Pupil Classroom Ratio|1188|108_._807<br>(1_._710)|1188|108_._229<br>(1_._683)|0_._578|
|Pupil Teacher Ratio|1196|80_._684<br>(1_._037)|1193|79_._708<br>(1_._122)|0_._976|
|Enrollment Grades 1-4|1197|608_._030<br>(12_._275)|1194|613_._517<br>(13_._909)|_−_5_._487|
|Enrollment Grades 5-8|1197|295_._145<br>(8_._354)|1194|307_._861<br>(9_._957)|_−_12_._716|
|Repetition Rate|1185|0_._259<br>(0_._003)|1181|0_._253<br>(0_._003)|0_._006|
|Dropout Rate|1185|0_._048<br>(0_._002)|1181|0_._044<br>(0_._002)|0_._004|

_Notes_ : Indicators extracted from EMIS 2017. The value displayed for t-tests are the differences in the means across the groups. Robust standard errors are reported in the second row (in parentheses). ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

Table 2: Balance: Impact Evaluation Sample: Treatment vs Control

|Variable|N|(1)<br>Control<br>Mean/SE|N|(2)<br>Treatment<br>Mean/SE|T-test<br>Diference<br>(1)-(2)|
|---|---|---|---|---|---|
|Number of Toilets|184|14_._973<br>(0_._734)|202|13_._267<br>(0_._530)|1_._705_∗_|
|Pupil Classroom Ratio|182|107_._964<br>(4_._526)|201|105_._360<br>(4_._364)|2_._604|
|Pupil Teacher Ratio|184|75_._928<br>(1_._984)|201|77_._714<br>(2_._343)|_−_1_._786|
|Enrollment Grades 1-4|184|618_._212<br>(31_._354)|202|577_._119<br>(27_._025)|41_._093|
|Enrollment Grades 5-8|184|316_._685<br>(21_._046)|202|285_._287<br>(17_._675)|31_._398|
|Repetition Rate|184|0_._258<br>(0_._008)|200|0_._248<br>(0_._008)|0_._011|
|Dropout Rate|184|0_._048<br>(0_._005)|200|0_._049<br>(0_._005)|_−_0_._001|

_Notes_ : Indicators extracted from EMIS 2017. The value displayed for t-tests are the differences in the means across the groups. Robust standard errors are reported in the second row (in parentheses). ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

27
Table 3: Differential Dropout and Attrition

||(1)<br>% Students Dropped Out<br>_b/se_|(2)<br>% Students Non-Attritted<br>_b/se_|(3)<br>% Teachers Non-Attritted<br>_b/se_|
|---|---|---|---|
|Treatment|_−_2_._841|0_._176|_−_0_._943|
||(2_._14)|(2_._00)|(2_._11)|
|Constant|34_._264_∗∗∗_|84_._833_∗∗∗_|80_._866_∗∗∗_|
||(1_._52)|(1_._45)|(1_._53)|
|Observations|337|337|337|
|R-sqr|0_._005|0_._000|0_._001|

Data Source: MLSS 2021. Shares of longitudinal sample. Analytical sample restricted to MLSS schools observed in baseline and endline. Robust standard errors are reported in the second row (in parentheses). ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

Table 4: Impact of the Intervention on Test Scores

||(1)|(2)|(3)|
|---|---|---|---|
||Math Score|English Score|Total Score|
|Treatment|12_._979_∗∗_|8_._566|10_._443_∗_|
||(6_._094)|(6_._355)|(5_._940)|
|Constant|339_._098_∗∗∗_|325_._052_∗∗∗_|305_._597_∗∗∗_|
||(9_._468)|(10_._726)|(10_._788)|
|Observations|4363|4363|4363|
|R-sqr|0_._113|0_._123|0_._151|

_Notes:_ This table looks at the impact of the school leadership on Test Scores (columns 1-3) at the student level. The outcome variables are: Math Knowledge Score (column 1); English Knowledge Score (column 2); Total Knowledge Score (column 3). Analytical sample restricted to MLSS schools observed in baseline and endline. All regressions are adjusted by the value of the outcome variable observed at MLSS baseline. First row of the table shows point estimate coefficients and statistical significance. Robust standard errors (clustered at the school level) are reported in the second row (in parentheses). ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

28
Table 5: Impact of the Intervention on Student Progression

|||Repetition Rate|||Dropout Rate||
|---|---|---|---|---|---|---|
||(1)<br>Lower Primary|(2)<br>Upper Primary|(3)<br>Overall Primary|(4)<br>Lower Primary|(5)<br>Upper Primary|(6)<br>Overall Primary|
|Treatment|_−_2_._249_∗_|_−_0_._798|_−_1_._773|0_._308|0_._181|0_._087|
||(1_._339)|(1_._393)|(1_._170)|(0_._826)|(0_._823)|(0_._744)|
|Constant|25_._108_∗∗∗_|18_._627_∗∗∗_|21_._039_∗∗∗_|5_._210_∗∗∗_|6_._614_∗∗∗_|5_._922_∗∗∗_|
||(1_._494)|(1_._466)|(1_._370)|(0_._628)|(0_._739)|(0_._648)|
|Observations|385|380|385|385|380|385|
|R-sqr|0_._041|0_._051|0_._072|0_._018|0_._028|0_._024|

_Notes:_ This table looks at the impact of the school leadership on Repetition Rate (columns 1-3) and Dropout Rate (columns 4-6) at the school level. All regressions are adjusted by the value of the outcome variable observed at MLSS baseline. First row of the table shows point estimate coefficients and statistical significance. Robust standard errors are reported in the second row (in parentheses). ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

Table 6: Impact of the Intervention on Test Scores (Heterogeneity by PTR in lower primary at baseline)

||(1)<br>Math Score|(2)<br>English Score|(3)<br>Total Score|
|---|---|---|---|
|Treatment|7_._739|_−_0_._813|3_._585|
||(9_._012)|(9_._236)|(8_._757)|
|Below Median PTR in Lower Primary|11_._592|4_._262|7_._970|
||(8_._737)|(9_._288)|(8_._566)|
|Heterogeneous Treatment Impact|12_._759|23_._384_∗_|17_._700|
||(12_._955)|(13_._850)|(12_._788)|
|Constant|489_._581_∗∗∗_|493_._805_∗∗∗_|491_._809_∗∗∗_|
||(6_._152)|(6_._375)|(6_._014)|
|Observations|4346|4346|4346|
|R-sqr|0_._008|0_._008|0_._009|

_Notes:_ This table looks at the heterogeneous impact of the school leadership on student level Test scores (columns 1-3) with respect to school-level Lower Primary (Grades 1-4) Pupil-Teacher Ratio (PTR) at baseline. The heterogeneous treatment impact is the estimated coefficient for the value of the treatment effect interacted with the variable of interest. Analytical sample restricted to MLSS schools observed in baseline and endline. All regressions are adjusted by the value of the outcome variable observed at MLSS baseline. The outcome variables are: Math Knowledge Score (column 1); English Knowledge Score (column 2); Total Knowledge Score (column 3). All regressions include clustered (at school level) standard errors. Robust standard errors (clustered at the school level) are reported in the second row (in parentheses). ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

29
Table 7: Impact of the Intervention on Test Scores (Heterogeneity by gender)

||(1)<br>Math Score|(2)<br>English Score|(3)<br>Total Score|
|---|---|---|---|
|Treatment|14_._361_∗_|10_._796|12_._670_∗_|
||(7_._620)|(8_._244)|(7_._520)|
|Female|_−_9_._922|1_._668|_−_3_._772|
||(6_._093)|(6_._010)|(5_._606)|
|Heterogeneous Treatment Impact|_−_0_._900|_−_1_._166|_−_1_._316|
||(8_._322)|(8_._304)|(7_._719)|
|Constant|500_._340_∗∗∗_|495_._023_∗∗∗_|497_._635_∗∗∗_|
||(5_._129)|(5_._556)|(5_._023)|
|Observations|4363|4363|4363|
|R-sqr|0_._004|0_._002|0_._003|

_Notes:_ This table looks at the heterogeneous impact of the school leadership on student level Test scores (columns 1-3) with respect to student gender. The heterogeneous treatment impact is the estimated coefficient for the value of the treatment effect interacted with the variable of interest. The outcome variables are: Math Knowledge Score (column 1); English Knowledge Score (column 2); Total Knowledge Score (column 3). Analytical sample restricted to MLSS schools observed in baseline and endline. All regressions are adjusted by the value of the outcome variable observed at MLSS baseline. Robust standard errors (clustered at the school level) are reported in the second row (in parentheses). ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

Table 8: Impact of the Intervention on Test Scores (Heterogeneity by student age at baseline)

||(1)<br>Math Score|(2)<br>English Score|(3)<br>Total Score|
|---|---|---|---|
|Treatment|13_._355_∗∗_|9_._174|11_._226_∗_|
||(6_._652)|(6_._860)|(6_._396)|
|Below Median BL Student age|64_._247_∗∗∗_|72_._024_∗∗∗_|67_._681_∗∗∗_|
||(6_._300)|(7_._138)|(6_._123)|
|Heterogeneous Treatment Impact|_−_1_._390|0_._911|_−_0_._343|
||(8_._830)|(10_._129)|(8_._743)|
|Constant|477_._676_∗∗∗_|476_._505_∗∗∗_|477_._379_∗∗∗_|
||(4_._319)|(4_._644)|(4_._229)|
|Observations|4183|4183|4183|
|R-sqr|0_._055|0_._070|0_._073|

_Notes:_ This table looks at the heterogeneous impact of the school leadership on student level Test scores (columns 1-3) with respect to student age at baseline. The heterogeneous treatment impact is the estimated coefficient for the value of the treatment effect interacted with the variable of interest. The outcome variables are: Math Knowledge Score (column 1); English Knowledge Score (column 2); Total Knowledge Score (column 3). Analytical sample restricted to MLSS schools observed in baseline and endline. All regressions are adjusted by the value of the outcome variable observed at MLSS baseline. Robust standard errors (clustered at the school level) are reported in the second row (in parentheses). ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

30
Table 9: Impact of the Intervention on Test Scores (Heterogeneity by total score at baseline)

||(1)<br>Math Score|(2)<br>English Score|(3)<br>Total Score|
|---|---|---|---|
|Treatment|4_._242|4_._126|4_._064|
||(7_._645)|(7_._954)|(7_._219)|
|Below Median BL Total Knowledge Score|_−_82_._778_∗∗∗_|_−_82_._831_∗∗∗_|_−_82_._374_∗∗∗_|
||(5_._827)|(5_._884)|(5_._223)|
|Heterogeneous Treatment Impact|16_._327_∗_|10_._494|13_._547|
||(8_._938)|(8_._988)|(8_._258)|
|Constant|539_._553_∗∗∗_|540_._607_∗∗∗_|540_._001_∗∗∗_|
||(4_._912)|(5_._170)|(4_._585)|
|Observations|4203|4203|4203|
|R-sqr|0_._085|0_._091|0_._103|

_Notes:_ This table looks at the heterogeneous impact of the school leadership on student level Test scores (columns 1-3) with respect to total score at baseline. The heterogeneous treatment impact is the estimated coefficient for the value of the treatment effect interacted with the variable of interest. The outcome variables are: Math Knowledge Score (column 1); English Knowledge Score (column 2); Total Knowledge Score (column 3). Analytical sample restricted to MLSS schools observed in baseline and endline. All regressions are adjusted by the value of the outcome variable observed at MLSS baseline. Robust standard errors (clustered at the school level) are reported in the second row (in parentheses). ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

31
Table 10: Impact of the Intervention on several mechanism indices

||(1)<br>Resource utilization|(2)<br>Teacher management|(3)<br>Teacher efort|(4)<br>Teacher practices|(5)<br>Remedial classes|(6)<br>School culture|
|---|---|---|---|---|---|---|
|Treatment|0_._088|0_._148|0_._031|_−_0_._003|0_._193_∗_|_−_0_._116|
||(0_._094)|(0_._101)|(0_._103)|(0_._102)|(0_._102)|(0_._102)|
|Constant|_−_0_._046|_−_0_._099|_−_0_._009|_−_0_._057|_−_0_._085|0_._007|
||(0_._074)|(0_._080)|(0_._082)|(0_._076)|(0_._082)|(0_._113)|
|Observations|386|386|386|386|386|386|
|R-sqr|0_._144|0_._039|0_._007|0_._024|0_._022|0_._004|

_Notes:_ This table looks at the impact of the school leadership on summary indices of intermediate outcomes at school level. Indices constructed using the generalized least-square method proposed by Anderson (2018: Journal of the American Statistical Association 103: 1481–1495). All regressions are adjusted by the value of the outcome variable observed at MLSS baseline. Significant findings are robust to multiple hypothesis testing based on FWER Romano-Wolf p-values from 2000 bootstrapped replications. Robust standard errors are reported in the second row (in parentheses). ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

_Resource utilization_ includes indicators relating to lower primary PTRs, the ratio of PTRs between lower and upper primary, and lower primary pupil-textbook ratios.

_Teacher management_ , includes frequency of observation of teachers by Head Teachers and of use of rewards for teacher performance.

_Teacher effort_ includes teacher attendance and presence in the classroom, assigning and marking of homework, teacher job satisfaction, and time on task.

_Teaching practices_ includes whether teachers asked students questions, employed teaching aids or local materials to help deliver lessons, or displayed instructional materials in the classroom.

_Remedial classes_ includes the frequency of remedial classes and share of grades in which they are offered. _School cultures_ includes measures of the incidence of physical and verbal bullying by teachers and students. See Appendix A for full details of intermediate indicators.

Table 11: Impact of the Intervention on the components of the Remedial Classes mechanism

||(1)<br>School ofers<br>remedial classes|(2)<br>Remedial Classes<br>Average Frequency|(3)<br>Remedial Classes<br>(Share of Grades)|(4)<br>Free<br>Remedial Classes<br>(Share of Grades)|(5)<br>Lower Primary<br>Remedial Classes<br>(Share of Grades)|(6)<br>Upper Primary<br>Remedial Classes<br>(Share of Grades)|
|---|---|---|---|---|---|---|
|Treatment|0_._042|0_._655_∗∗_|5_._325_∗∗_|5_._220_∗∗_|3_._884|6_._602_∗_|
||(0_._026)|(0_._328)|(2_._550)|(2_._519)|(2_._970)|(3_._678)|
|Constant|0_._900_∗∗∗_|4_._518_∗∗∗_|54_._397_∗∗∗_|53_._614_∗∗∗_|87_._990_∗∗∗_|15_._364_∗∗∗_|
||(0_._046)|(0_._241)|(2_._003)|(1_._989)|(2_._412)|(2_._638)|
|Observations|386|386|386|386|386|386|
|R-sqr|0_._009|0_._022|0_._015|0_._015|0_._006|0_._015|

_Notes:_ This table looks at the impact of the school leadership on prevalence of remedial classes at the school level. All regressions are adjusted by the value of the outcome variable observed at MLSS baseline. Significant findings are robust to multiple hypothesis testing based on FWER Romano-Wolf p-values from 2000 bootstrapped replications. Robust standard errors are reported in the second row (in parentheses). ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

32