metadata: authors: 0: author: Filmer,Deon P.
1: author: Habyarimana,James Paul
2: author: Sabarwal,Shwetlena
count: Tanzania
docna: 0: docna: Teacher Performance-Based Incentives and
            Learning Inequality
docty: Policy Research Working Paper
subtopic: Effective Schools and Teachers,Educational Institutions & Facilities,Educational Sciences,Public Sector Administrative & Civil
            Service Reform,De Facto Governments,Public Sector Administrative and Civil
            Service Reform,Administrative & Civil Service Reform,Democratic Government
teratopic: Education,Social Protections and Labor,Governance
abstracts: cdata!: This study evaluates the impacts of
            low-cost, performance-based incentives in Tanzanian
            secondary schools. Results from a two-phase randomized trial
            show that incentives for teachers led to modest average
            improvements in student achievement across different
            subjects. Further, withdrawing incentives did not lead to a
            "discouragement effect" (once incentives were
            withdrawn, student performance did not fall below
            pre-baseline levels). Rather, impacts on learning were
            sustained beyond the intervention period. However, these
            incentives may have exacerbated learning inequality within
            and across schools. Increases in learning were concentrated
            among initially better-performing schools and students. At
            the same time, learning outcomes may have decreased for
            schools and students that were lower performing at baseline.
            Finally, the study finds that incentivizing students without
            simultaneously incentivizing teachers did not produce
            observable learning gains.
geo_reg_key: 517191
repnme: repnme: Teacher Performance-Based Incentives and
            Learning Inequality
display_title: Teacher Performance-Based Incentives
            and Learning Inequality
listing_relative_url: /research/2020/09/32386251/teacher-performance-based-incentives-learning-inequality
topicv3: Teachers Management m1326119 1523,Teacher Pay and Incentives m1326551 1530,Individual Performance Management m1326767 2058,Education m1327048 470,Teacher Performance m1327509 1531
publishtoextweb_dt: 2020-09-08T00:00:00Z
historic_topic: Education,Social Protections and Labor,Governance
origu: Development Research Group (DECRG)
no_of_pages: 35
author_keys: 0: author_key: 97345
1: author_key: 615101
2: author_key: 950739
model_extractions: Policy Research Working Paper 9382 Teacher Performance-Based Incentives and Learning Inequality _Deon Filmer James Habyarimana Shwetlena Sabarwal_ Development Economics Development Research Group & Education Global Practice August 2020
Policy Research Working Paper 9382

# **Abstract**

This study evaluates the impacts of low-cost, performance-based incentives in Tanzanian secondary schools. Results from a two-phase randomized trial show that incentives for teachers led to modest average improvements in student achievement across different subjects. Further, withdrawing incentives did not lead to a “discouragement effect” (once incentives were withdrawn, student performance did not fall below pre-baseline levels). Rather, impacts on learning were sustained beyond the intervention period. However, these incentives may have exacerbated learning inequality within and across schools. Increases in learning were concentrated among initially better-performing schools and students. At the same time, learning outcomes may have decreased for schools and students that were lower performing at baseline. Finally, the study finds that incentivizing students without simultaneously incentivizing teachers did not produce observable learning gains.

This paper is a product of the Development Research Group, Development Economics and the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank. org/prwp. The authors may be contacted at dfilmer@worldbank.org, jph35@georgetown.edu, and ssabarwal@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Teacher Performance-Based Incentives and Learning Inequality**

Deon Filmer The World Bank James Habyarimana Georgetown University Shwetlena Sabarwal The World Bank Keywords: Student achievement; Teacher Incentives; Student Incentives; Learning outcomes; Inequality JEL Codes: I21; I28; J45 Acknowledgements: We thank the staff of the Tanzania Ministry of Education, Science and Technology for extensive discussions on program design and implementation. We are grateful for comments from Samer Al Samarrai, Alaka Holla, Tobias Linden, Ezequiel Molina, and participants at the 2019 Research In Improving Systems of Education (RISE) conference. Tummaniel Ngowi, Sana Khan, Tiffany Tong, and Ravinder Gera provided excellent field-based support. Avichal Mahajan, Ali Hamza, Olivia Bisel, Frances Chen, Rachel Leeds, and Chinmaya Holla were superb research assistants. Financial support from the Strategic Impact Evaluation Fund, Multi-Donor Education and Skills Fund, Strategic Research Program, and RISE is gratefully acknowledged. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represent.
# **1. Introduction**

Given the structure of their contracts and their political influence, teachers, especially in low- and middle-income countries, are only weakly extrinsically incentivized to exert effort towards student learning (Hanushek 1996, Lockheed and Verspoor 1991, World Bank 2004, 2018). Performance-based incentives have emerged as a potential way to align teacher effort towards student learning. This is motivated by two features of the education sector. First, observable teacher characteristics such as experience and qualifications, which are the main determinants of salaries in most school systems, are weak predictors of teacher effectiveness in producing student learning (Rivkin, Hanushek and Kain 2005; Aaronson, Barrow and Sander 2007). Second, using direct personal incentives might be better than intervening in the educational process directly because policy makers may not know the best means of improving education given the heterogeneous attributes of students and teachers. Providing incentives tied to student performance allows schools and teachers to choose the best means to improve performance given their circumstances.

Data, albeit from primary schools, suggest that there are significant margins to increase teacher effort in Tanzania. In 2014, students were being taught for about 2 hours and 47 minutes per day out of 5 hours and 56 minutes of scheduled teaching time (Martin and Wane 2016). There were no major differences in learning time across urban and rural schools. Like in many other Sub-Saharan African countries, participation rates at the secondary school level have increased dramatically in Tanzania, for example the lower secondary gross enrollment rate increased from 20 percent in 2000 to close to 60 percent in 2016 (Bashir et al. 2018). At the same time there have been persistent concerns with poor learning outcomes at both the primary and secondary levels, which have, in turn, been linked to low levels of subject knowledge and low effort among teachers (Bold et al. 2017a; 2017b).

Our study’s main goal is to assess whether addressing teacher effectiveness through fiscally and politically viable low-cost performance-based incentives can be sufficient to make a difference to student learning outcomes. We further investigate five aspects of such an intervention. These aspects emerged as areas of concern expressed by Tanzanian education policymakers<sup>1</sup> in the context of discussions around the

> 1 The research team discussed the evaluation design with the Permanent Secretaries (PS) in Tanzania Ministry of Education and Vocational Training (MoEVT) (which is the antecedent of the current Ministry of Education, Science and Technology)  and President's Office, Regional Administration and Local  Government. The team worked <u>closely with MOEVT focal points assigned by MoEVT PS.</u>

2
potential benefits and drawbacks of introducing teacher incentives. First, we evaluate the impact of withdrawing the incentives with the goal of establishing whether one might find a discouragement effect. Second, we assess the durability of the impacts by following students beyond the intervention period. Third, we document heterogeneity in the impact of the teacher incentives. Fourth, we explore the pathways through which the effects might work. Last, we contrast incentives for teachers with those for students.

We study these questions through a two-phase randomized research design focusing on Grade 10 teachers and students in about 400 schools in Tanzania. Study schools were randomly assigned into one of a set of treatment arms in each phase of the program. In phase 1, schools were randomly assigned to either a control group, an arm implementing teacher incentives, or an arm implementing both teacher and student incentives. Teachers in schools assigned to the teacher incentive arm “competed” with teachers in approximately 6-12 other comparable schools—and teachers with the greatest value added in terms of student test scores received an award. Students assigned to the student incentive arm “competed” with students in the same school—and students with the greatest increase in terms of test scores received an award. In phase 2, student and teacher incentives programs were implemented but with a twist. First, teacher incentive schools in phase 1 were randomly split into schools that would implement teacher incentives for the second year and schools where those incentives were withdrawn. Second, the control group from phase 1 was randomly split into a pure control group and a student incentives-only group. So overall, schools were randomly assigned to either a control group, an arm implementing teacher incentives for a second year, an arm in which teacher incentives were discontinued, and an arm implementing student incentives alone. The design is summarized in Figure 1.<sup>2</sup> Our main finding is that teacher incentives led to a modest average increase in student learning outcomes. In phase 1 these were only found when combined with student incentives, in phase two these were found for the simple teacher incentive intervention. The effect sizes we find range from 0.09 to 0.18 standard deviations (sd), depending on the phase and the test score (Mathematics, English, or Kiswahili) used to measure impacts. We do not find any evidence of a discouragement effect when teacher incentives were removed: in fact, the effects remain positive, although they are generally smaller and no longer statistically significant. Repeating the teacher incentives intervention in phase 2 resulted in similar impacts on student learning as in phase 1, suggesting that there was neither learning nor fatigue associated with the second round of teacher incentives. Next we show that learning impacts persisted. Specifically, when we follow-up with the phase 2 cohort of students more than one year after the intervention—when they are no

> 2 In Tanzania, teachers are typically linked to grades (not student cohorts). Since, we target the same grades in Phases 1 and 2, by design, the same teachers were involved in each phase. However, teacher turnover in Tanzania is high, therefore there was some change in the teacher groups between phases. Our school level estimates suggest that two thirds of teachers remain in the school across both phases of the study.

3
longer the students of an incentivized teacher and when all incentives have been completely withdrawn from the schools—we find effect sizes to be statistically significant and on the order of 0.1 sd. The last of the average effects we document are that learning outcomes did not increase as a result of student incentives when these incentives were implemented on their own (phase 2). When combined with teacher incentives (phase 1) they were more generally associated with increases in learning outcomes.

Beyond establishing modest average impacts, our second main finding is that there was substantial heterogeneity in learning impacts from the teacher incentives. We find this both across schools—learning impacts were concentrated in schools that had higher baseline test scores, and across students, learning impacts were largest for students in the higher (conditional) quantiles of the test score distribution—with effect sizes reaching as high as 0.4 standard deviation for some groups of students. At the same time, the impact in initially poorly performing schools was close to zero and sometimes negative for students in the lower (conditional) quantiles of the test score distribution (although not always statistically significantly so). Both dimensions of heterogeneity (across and within schools) go in the direction of exacerbating inequalities. This suggests that while, in this case, teacher incentives may have had a positive average impact, the benefits to students in terms of learning were very unequally distributed.<sup>3</sup> We also find that, in phase 2, student incentives implemented on their own had a positive impact on student test scores in schools that had higher baseline scores (but not in other schools).

Our paper contributes to two main literatures, namely those on teacher and student incentives. Empirical evidence on the effectiveness of teacher incentives is mixed.<sup>4</sup> Positive impacts on learning, with effects typically on the order of between 0.15 to 0.3 sd, have been found in randomized evaluations in a number of low-income countries (Glewwe et al. 2010 in Kenya; Loyalka et al. 2019 in China; Gilligan et al. 2019 in Uganda, Duflo et al. 2012, and Muralidharan and Sundararaman 2011a, 2011b in India).<sup>5</sup> In primary schools in Tanzania, Mbiti et al. (2019) found that teacher incentives led to a 0.21 sd increase in student scores after two years on the test used to set the awards but no impact on a separately administered test; when combined with (non-incentive-based) school grants, they found statistically significant impacts of 0.36 sd and 0.23 sd respectively on these tests.<sup>6</sup> In contrast, no impacts were found in an intervention that linked bonuses to student exam scores in Pakistan (Barrera-Osorio and Raju 2017). The finding of no

> 3 The observed teacher incentives or incentives withdrawn impacts in phase 2 are the average effects over groups that had received different treatments in phase 1 (teacher incentives only and teacher and student incentives).  The study design is not powered to detect potential differences by phase 1 assignment.

> 4 See Glewwe and Muralidharan (2016) for a recent review of the impact of teacher incentives for developing countries.

> 5 Positive and significant impacts are also seen in quasi-experimental studies in Israel (Lavy 2002, 2009).

> 6 In an evaluation of a pay for performance program in Austin, Texas, Balch and Springer (2015) find that the program is associated with positive student test score gains in both math and reading during the initial year of implementation but do not find any additional growth in the second year.

4
impacts has been documented in a number of high-income settings (Glazerman and Seifullah 2012; Springer et al. 2012 in the United States), and some have even found negative impacts (Martins 2009 in Portugal; Fryer 2013 in New York City, Atkinson et al. 2009 in England).

Recent meta-studies of interventions aimed at increasing learning outcomes in primary schools in lowand middle-income countries conclude that teachers do indeed respond to incentives (Evans and Popova 2016, McEwan 2015), especially in low-accountability settings (Ganimian and Murnane 2016). Examining the mixed evidence more closely suggests that impacts are likely to depend on context, especially on the existing margins for improving teacher effort. Incentives are likely to be more effective when teachers can determine and implement low-cost actions to improve student learning. This is likely to be the case in developing countries, where teacher absenteeism is high and time on task is low (Chaudhury et al. 2006, Bold et al. 2017b). Impacts also depend on the details of the design features of the incentive program, for example whether the incentives are for groups versus individuals, or whether rewards are financial versus non-financial (Bruns, Filmer and Patrinos 2011) or how performance is measured (Barlevy and Neal 2012).

A related literature focuses on perverse impacts of teacher incentives. This includes “teaching to the test” which was documented in Kenya (Glewwe et al. 2010)—although in another context (India) positive impacts spilled over to non-incentivized subjects (Muralidharan and Sundararaman 2011a). Much of this literature has focused on cheating, which has been documented in both middle- and high-income contexts (Behrman et al. 2015 in Mexico; Jacob and Levitt 2003 in the United States). Neal (2008) discusses a variety of ways in which schemes that link teacher incentives to student achievement could potentially be corrupted (also see Cullen and Reback 2006, Jennings and Beveridge 2009, Figlio, 2005 and Jacob 2005, 2007).

The literature on what happens when teacher incentives are withdrawn or on the heterogeneity in impacts is relatively thin. On the first, Jinnai (2016) found that in North Carolina (USA), where the state government first reduced and finally repealed its teacher incentive program, student achievement at the lowest-performing schools significantly decreased. On the second, Bacache‐Beauvallet (2006) argues theoretically why incentives would increase inequality, while Chang et. al (2020) evaluate a teacher incentive program that explicitly rewarded performance increases of initially poorer-performing students, and indeed found larger impacts for these students.

The second literature we contribute to is that on student incentives where empirical evidence from lowor middle-income settings is thin and inconclusive. Positive impacts have been documented in Kenya 5
(Kremer et al. 2007) and Benin (Blimpo 2014).<sup>7</sup> On the other hand, Li et al. (2010) found that a cash incentive linked to performance led to no measurable impacts on learning outcomes in China.<sup>8</sup> Evidence on student incentives programs from high-income countries is likewise mixed, finding no impacts overall (Fryer 2011 in three US cities); inconsistent results by gender of the student (Angrist and Lavy 2009 in Israel; Angrist et al. 2009 in Canada) or by subject (Bettinger 2012 in Ohio); positive impacts for students close to meeting the achievement standard in the United States (Levitt et al. 2016). Jackson (2010) found that combining teacher and student incentives had a significant impact on exam performance in Texas (USA).

There is only one study, to our knowledge, that directly compares teacher and student incentives (Behrman et al. 2015). The study experimentally contrasts the impact of three different performance incentive schemes in Mexican high schools and found that the largest impacts were for a “complete” package of individual and group incentives to students, teachers and school administrators, smaller impacts for individual student incentives only, and no impacts for the individual teacher incentives arm.

Our paper is organized as follows. Section 2 provides background information on the incentive program and the context of the study, experimental design, data, and estimation strategy; Section 3 presents the main results. Section 4 concludes.

# **2. Context, Experimental Design, Data, and Estimation Strategy**

The program we evaluate was implemented across a sample of lower secondary schools in three regions of Tanzania.<sup>9</sup> The intervention focuses on students in Grade 10 (Form 3, the third of four years of lower secondary school). The context is one in which teacher compensation and promotion are largely unrelated to performance in terms of student learning outcomes. Procedures for teacher recruitment and firing are

> 7 A study from Cambodia showed that transfers based on baseline performance (not linked to subsequent effort) had positive impacts on learning (Barrera-Osorio and Filmer 2016).

> 8 In an experiment in non-formal schools in Indian slums, a reward scheme for attending a target number of school days increased average attendance when the scheme was in place. However, among students with low baseline attendance, the incentive lowered post-incentive attendance and test scores—pointing to unintended negative consequences of the intervention (Visaria et al. 2016).

> 9 These are Lindi, Morogoro, and Shingyanga. Due to re-districting in Shinyanga during the course of the study, some schools were re-classified into Geita and Simiyu. However, for the purposes of this study, schools retain their 2012 regional classification. These three regions represented roughly 11 percent of Tanzania’s population in 2012. They were selected based on government guidance to reflect intermediate levels of socio-economic development across Tanzania. In the 2015 HDI index ranking of Tanzania’s 26 regions (with Rank 1 being best performing and Rank 26 being the worst performing), Lindi ranked 12, Morogoro 15, and Shinyanga 19. They also reflect high vs. low levels of private school penetration. In Morogoro, 21 percent of its total secondary schools were private in 2012 and in Shinyanga and Lindi this share was 8 percent and 6 percent, respectively.

6
complicated and lengthy. In addition, monitoring and supervision of teacher performance by school inspectors is weak and school management committees have little or no oversight over teachers (United Republic of Tanzania 2008). While improvements in the management and supervision of teachers are longrun goals, the objective of this study was to determine whether, in the short run, providing performancebased incentives to teachers and students could boost learning outcomes. Because fiscal constraints and political pressures would make it hard to scale up more high-powered incentives, the decision was made to use a relatively low-stakes and low-cost approach to incentives.<sup>10</sup>

# **2.1 Experimental design**

This research was initiated in 2013 and comprises of two phases: phase 1 covered the 2013 academic year (January-December) while phase 2 covered the 2014 academic year. Around 420 secondary schools (public and private) in three regions of Tanzania—one in the north-western part of the country (Shinyanaga) and two in the south-east (Lindi and Morogoro)—were sampled. The interventions were restricted to Grade 10 students and teachers and focused on student performance in Math, English, and Kiswahili. Because most students would have passed from Grade 10 to 11 between 2013 and 2014, the cohort of students in phase 1 is different from that in phase 2.

The full experimental two-phase design is summarized in Figure 1. In phase 1, schools were randomly assigned to one of two treatment arms or a control arm. In the first treatment arm teachers were eligible to receive performance incentives (TI), in the second treatment arm both teachers and students were eligible to receive incentives (TI+SI). The control group received nothing. In phase 2, assignments to treatment groups were based on innovations to phase 1 treatments, to best answer policy maker’s questions. Specifically, phase 1 control was randomly assigned to one of two phase 2 arms: a pure control and a student-incentives only arm (SI). In addition, phase 1 TI and TI+SI arms (together) were randomly divided into two phase 2 arms. In the first of these, teacher incentives were offered again (TI) and in the second, the teacher incentive program was discontinued (TI withdrawn). Student incentives were offered in neither of these groups.

The teacher incentive program was structured as follows. Schools were divided into relatively homogeneous groups within which teachers would “compete,” based on geographic sub-region and performance at baseline (6-12 schools per group). The objective of restricting the set of comparators to similar schools was to help ensure that all teachers believed that they had a reasonable chance of winning

> 10 See the discussion in Grindle (2004) on political pressures.  An additional reason to use non-financial incentives is because these may crowd out intrinsic motivation (List, Livingston and Neckermann 2018; also see discussion in Benabou and Tirole 2003).

7
an award. Awards for teachers were based on annual average gain in the score of the teacher’s students between baseline and endline tests within the same phase.<sup>11</sup> Gains were expressed as percentage-point increases in the score on the test, which was marked out of 100. The curriculum-based tests were designed specifically for this intervention and were administered at the beginning and end of the school year. Within each group of 6-12 schools, the top three teachers in each of Math, English, and Kiswahili received an inkind award. These were distributed at region-wide public ceremonies chaired by high-ranking district and regional officers. Awards included smart phones, book vouchers, certificates, and medals with an overall value of roughly $190 for first-place teachers, $130 for second-place teachers, and $110 for third-place teachers.<sup>12</sup> The student incentives were designed as within-school competitions. Within each school, students achieving the highest annual gains in the average score on the tests received an award. Again, gains were calculated in percentage points on the study-specific standardized curriculum-based test, which was marked out of 100. For each eligible school assigned to this arm, the top three students in Math, English, and Kiswahili were awarded. Awards, which included book vouchers, certificates, and medals, were distributed at school-based ceremonies.

The total annual cost per year for the teacher incentive program was approximately US$ 442, and that for the student incentive program was approximately US$ 92. This is excluding student assessment costs under the assumption that the awards can be tied to regular system-level student assessments.<sup>13</sup>

# **2.2  Sampling**

Sampling of schools was done using a complete list of public and private secondary schools in the three regions.<sup>14</sup> All wards with private schools were automatically included in the sample. After this step was completed, remaining wards were selected randomly, with the goal of reaching a sample size of over 400

> 11 The announcement for the awards included a note that students who were absent on the day of the endline test would get a score of zero. Students who were absent at baseline but present at endline would receive the average baseline score for the school.

> 12 Analysis of Tanzania’s 2014 Labor Force Survey suggests that median monthly earnings of secondary school teachers are on the order of $376. This is reported as PPP$934 in Evans, Yuan, and Filmer (2020) based on a survey median of Tanzanian shillings 622,434.  $376 is calculated on the basis of an exchange rate of 1653 shillings per dollar as reported in World Development Indicators for 2014.

> 13 Intervention costs increase by approximately US$ 500 per school, per year if the cost of designing, administering, invigilating, marking, and data-entry of student assessments is factored-in.

> 14 The sample for the study was constructed using complete secondary school lists for three regions in Tanzania – Lindi, Shinyanga, and Morgoro. The school list, disaggregated at the ward level, is derived by triangulating two administrative databases - EMIS 2011 and Inspectorate 2011 data. These data were augmented by a school mapping database created in 2013 as a part of the fieldwork for this research. We included all the schools for which we could find school performance data (Form 2 pass rates) for 2012. The final sample included 61 percent of all public secondary schools and 100 percent of all private secondary schools in the three regions.

8
schools and with the distribution of schools across regions being in proportion to their relative population size (using the 2002 census). The final sample consists of 62 schools in Lindi, 138 in Morogoro and 220 in Shinyanga.<sup>15,16</sup> Schools were initially randomly assigned to different treatment groups using a three-step process. First, for each region, a natural category (region, district, or combination of districts) with about 60 schools was constructed.<sup>17</sup> Second, within each natural category, we generated a ranking of school quality using Form II pass rates (derived from Inspectorate Data) and then we created three strata using terciles of the pass rate ranking.<sup>18</sup> Third, random assignment was undertaken as follows: (i) within each tercile, half of the schools were randomly assigned to receive teacher incentives with other half as control; (ii) within the teacher incentives arm, half of the schools were randomly assigned to teacher incentives only and the other half to teacher incentive plus student incentives; and (iii) within the control arm, half of the schools were randomly assigned to receive another treatment which turned out to be unfeasible and these schools were not visited again in Phase 1, with the other half assigned to have continual data collection during phase 1. This produces a sample with phase 1 assignments of roughly 25 percent to teacher incentives only, 25 percent to both teacher and student incentives, 25 percent to unfollowed control, and 25 percent to followed control.

In phase 2, half of the schools in which teacher incentives had been implemented (either with or without student incentives) were randomly selected to continue with the scheme, and half of the schools were selected to discontinue it. In addition, the control group from phase 1 (including the unfollowed arm) was randomly split into a pure control group and a student-incentives only group. This produces a sample with phase 2 assignments of roughly 25 percent to teacher incentives (repeated), 25 percent to teacher incentives withdrawn, 25 percent to student incentives, and 25 percent to control.

> 15 At the time that the sample was generated, a list of replacement schools was also generated from the population of schools. Schools were replaced for two reasons: (i) the school was no longer functioning or where duplicates were found; or (ii) the school either did not have a Form 3 cohort or had 10 Form 3 students or fewer. 16 At the time of Phase 1 Baseline (November 2012), the full impact evaluation sample of 409 schools included 206 control schools. These 206 schools were then sub-divided into two groups - 105 pure control schools which were not visited again in Phase 1 and 101 control schools which were visited in Phase 1. Consequently, in Phase 1 midline and endline, the total number of schools visited is 304 (409 – 105). In Phase 2, the 105 pure control schools were added back to the evaluation sample.   Between Phase 1 Baseline and Phase 2 Baseline, the school attrition was around 13 schools (409-396), due to school closures or school desire not to participate further in the impact evaluation.  During Phase 2 another 3 schools dropped out of the evaluation. The school sample is smaller for Phase 2 midline (385) because of the timing (September 2014) when some schools were found closed for examination preparation holiday. 17 For Lindi, there was no sub-division. The Morogoro region was divided into 3 such categories and Shinyanga was divided into 4

18 When Form II pass rates were not available, the value was predicted by using 1) Form IV pass rates or 2) baseline results. When no data were available, the schools were assigned to tercile 2.

9
# **2.3 Data**

Data collection began in November 2012 and lasted until September 2015 (Table 1). For each phase, three rounds of data were collected: baseline data at the start of the school year; midline data in the middle; and endline data at the end of the school year. Phase 1 baseline data were collected in November 2012February 2013, midline in August-September 2013, and endline in November 2013. Phase 2 data were collected at corresponding times in 2014. At baseline and endline, students were tested and teachers and head-teachers were surveyed. At midline, attendance data were collected for both students and teachers during unannounced visits. Teachers were observed teaching a class during the midline visits and their behaviors recorded along a number of dimensions using an instrument with 21 items across the following domains: how teachers interact with students; the materials they used during instruction; use of the blackboard; and assignment and review of homework.<sup>19</sup> After phase 2, post-endline data were collected in August-September 2015 during which the phase 2 cohort of students was re-tested only in math. Test scores used in our empirical analysis are all normalized to have a mean of 0 and a standard deviation of 1 for the control group in the relevant phase.

Around 83 percent of the schools in our sample are public schools; 92 percent are O Level only (go up to Grade 11). At time of the Phase 1 baseline (2013), around 36 percent of schools had electricity and almost all schools (99 percent) collected school fees. Around 56 percent of the schools in our sample have less than 10 permanent teachers; 11 percent have more than 20 permanent teachers. The majority of headteachers (85 percent) and teachers (79 percent) are male. Nearly 83 percent of teachers have more than secondary education and 90 percent have received some type of formal teacher training.

# **2.4 Estimation strategy**

We divide our empirical investigation into four main parts. We start by analyzing average impacts on student test scores in phases 1 and 2. Given the random assignment study design, our estimation of treatment impacts is straightforward. All of the estimates described below are intent-to-treat. We regress endline test scores _Testi1_e_ for phase 1 and _Testi2_e_ for phase 2 on the set of indicators capturing random assignment to a study arm. We control for a set of indicators to capture the stratified random assignment

> 19 For each phase, we conducted a separate exploratory factor analysis and retained the top four leading factors. In phase 1, the leading factors, which explain just under 38 percent of the variation across all items are shaped by items related to a) teacher-pupil interactions, b) use of materials such as maps, textbooks or equipment in instruction; c) use of the blackboard to copy lessons and illustrate examples, and d) assignment and review of homework. In phase 2, the four leading factors explain 36 percent of total variation and their dimensionality is explained by the same sets of items as the Phase 1 leading factors.

10
to arms (Region*PassTercile). We also adjust for an indicator that the school’s baseline average test score exceeds 50 percent (B50i). Our main specifications are therefore: Phase 1: Testi1_e = β1_0 + β1_1*TI_onlyi + β1_2*TI_SIi + δ 1_0*B50i +Σrjκ1_rjRegionr*PassTercilej + ε1_i (1) Next, we investigate heterogeneity in impacts using two approaches. First, in order to explore crossschool heterogeneity, we rely on machine learning to identify the covariates associated with treatment heterogeneity.<sup>20</sup> Guided by the machine learning results, we include an indicator variable for whether the baseline average score for the school was above 50 percent (B50i1) and interact that indicator with the indicators for each study arm. Roughly 30 percent of schools are above that threshold. The specifications we estimate are therefore: Phase 1: Testi1_e = β1_0 + β1_1*TI_onlyi + β1_2*TI_SIi + δ 1_0*B50i1 + δ 1_1*B50i1*TI_onlyi + Second, in order to explore cross-student heterogeneity, we re-estimate equations (1) and (2) using quantile regressions. Specifically, we estimate the impacts on students at the 10<sup>th</sup> , 25<sup>th</sup> , 50<sup>th</sup> , 75<sup>th</sup> , and 90<sup>th</sup> quantiles of the distribution of ε1/2_i, namely the test score conditional on all the variables capturing the

> 20 We use the _generalized random forests_ machine learning algorithm (Athey and Wager 2019) to identify the sources of variation in conditional average treatment effects. Observations are clustered at the school level with a minimum cluster size of 20 students. Across the three treatments, the most important variable driving treatment heterogeneity is the school average baseline average (aggregated across all three subject tests) with a variable importance between 24 and 70 percent. In particular, there is a sharp difference in treatment effects between schools below and above a baseline average score of 50 percent. In the full sample, other variables of importance include the 2012 Form 2 exam pass rate, baseline student performance and wealth. In the restricted teacher-student matched sample, class size and whether headteachers observe instruction are among the other top 10 most important variables.

11
research design for both Phase 1 and 2. In addition, we also re-estimate equations (3) and (4) using quantile regressions which allows us to explore both cross-school and cross-student heterogeneity.

In a third part of our empirical analysis we estimate whether the impacts are sustained beyond the intervention period. In order to do this, we re-estimate equations (2) and (4) replacing the dependent variable with the math test scores that were collected from Phase 2 students in late 2015—almost 9 months after they would have been covered by the intervention.

In the last part of our empirical analysis we explore various pathways through which impacts might have been generated. In order to do so, we re-estimate models (1)-(4) replacing the dependent variables with measures of teacher attendance, teacher pedagogical behaviors, and student attendance.

# **3. Results**

## **3.1 Balance at baseline**

Before turning to the analysis of impacts, we first establish that our experimental design is valid by examining the first moments of selected characteristics of schools, teachers, and students in control and treatment groups, using baseline data for phases 1 and 2 (Table 2). We report the p-value of an F-test corresponding to a null hypothesis that there is no difference across the three (in phase 1) and four (in phase 2) arms – clustered at the level of treatment assignment which is the school. In phase 1, the p-values for the test are all, with 1 exception out of the 23 variables, above 0.1 suggesting that the samples are very likely balanced. In phase 2, the p-values for all but 5 variables out of the 23 variables are above 0.1. We nevertheless believe that the samples are well balanced since the absolute magnitudes of the differences across these samples are small in magnitude. For example, the share of schools with electricity is 0.313 in the control group and 0.391 in the arm in which teacher incentives were withdrawn. The largest difference in magnitude is for the variable “headmaster rewards teachers who perform better” where the share is lowest in the student incentives arm (0.533) and highest in the teacher incentives arm (0.677). Importantly, the balance tests suggest that there are no statistically significant differences in the school average baseline test scores of students in the various study arms—either in phase 1 or in phase 2.

## **3.2 Average impacts on student achievement**

Impacts on student learning outcomes in Kiswahili, Math, English, as well as averaged across the three subjects, are reported in Table 3. In phase 1, we find positive and statistically significant learning impacts for the schools that received both teacher and student incentives. These impacts are in the range of 0.14 to 0.18 standard deviation. In phase 2, we find positive and significant learning impacts of teacher incentives 12
which are of a similar—albeit smaller—magnitude (0.09-0.14 sd). We are unable to detect any learning impacts of teacher incentives alone in phase 1 or of student incentives alone in phase 2. Notably, the impacts on learning persist even after teacher incentives are withdrawn. While these are only statistically significant for Kiswahili (column 5) the magnitude of the impacts are similar for the other subjects albeit more imprecisely measured (the effect sizes are all within the range of about 0.1 sd).<sup>21</sup> It is somewhat unclear why, in phase 1, teacher incentives only have an impact when combined with student incentives, and why, in phase 2, student incentives have no impact when implemented alone. One explanation could be the following. If extra teacher effort is required in order to generate impacts, then that would explain both the lack of impact from student incentives alone as well as the impact from combining them with teacher incentives. But why then did teacher incentives alone not have an impact in phase 1 but did in phase 2? It is possible that during phase 1 the combined intervention created more of an “event” in the school and the program as a whole was therefore more salient to teachers. With time, however, the salience of the teacher incentives, even alone, increased leading to the detectable impacts in phase 2. In addition, it may have taken some time for teachers to view the program as credible, again making it more salient in phase 2.

We take these results as suggesting that low stakes performance-based incentives for teachers have the potential for improving student learning, with modest average impacts. At the same time, student performance-based incentives alone are not likely to lead to appreciable impacts—although combined with teacher incentives they may have promise. Last, we do not find evidence suggesting that the withdrawal of teacher (and student) incentives led to a discouragement effect (in fact we see that impacts seem to persist— albeit with smaller magnitudes).

# **3.3 Heterogeneity in learning impacts**

Table 4 reports the results from estimating models that allow for impacts to differ by a school’s average test score at baseline. In phase 1 the results suggest that—while imprecisely measured—teacher incentives alone may have larger impacts on learning outcomes in the initially higher performing schools. The impact in initially poorly performing schools is close to zero and sometimes negative (although statistically insignificantly so), whereas for initially higher performing schools the impact is on the order of magnitude of about 0.04-0.1 sd higher (not significant). The results suggest differentiation in the effect of teacher and student incentives combined, especially for English, where the point estimate on the interaction is a very

> 21 Our school-level estimates suggest that two thirds of teachers remain in the school across both phases of the study. This estimate is consistent with considerable teacher turnover following a large secondary school construction program between 2006 and 2010. Across _all teachers,_ 22 percent of teachers leave between phase 1 and 2, while new teachers account for just over 38 percent.

13
substantial 0.45 sd, significant at the 10 percent level. Across all subjects, the average treatment effect differential is 0.2 sd, albeit imprecisely estimated.

The most striking results in Table 4 come from Phase 2. Here we find strong impacts of teacher incentives alone, but these are entirely concentrated in the schools that performed better at baseline. The impacts are statistically significant for all subjects, and the effect sizes are large (0.22 – 0.42 sd). For the average test score across the three subjects, teacher incentives alone led to an increase in learning of 0.33 sd (=0.345-0.019). While not statistically significantly different from 0, the size of the impacts of student incentives alone are also large in magnitude in the initially higher performing schools (with impacts on the order of 0.2 sd) suggesting heterogeneity for this intervention as well. Last, it is for these (initially) high performing schools that withdrawing teacher (and student) incentives seems to have had no negative effect—indeed the effect size for this group is similar to that in which teacher incentives were repeated.

We further investigate heterogeneity by estimating quantile regression models (Tables 5 and 6). The estimates for the size of the impact of teacher and student incentives in phase 1, and of teacher incentives in phase 2, becomes progressively larger as the reference quantile increases. In phase 1 they increase from 0.1 sd at the 10<sup>th</sup> percentile to 0.21 sd at the 90<sup>th</sup> percentile (Table 5 Columns 1-5); in phase 2 they increase from 0.02 sd (and statistically insignificant) at the 10<sup>th</sup> percentile to 0.17 sd (and statistically significant) at the 90<sup>th</sup> percentile (Table 6 Columns 1-5). This means that students whose (conditional) test scores were lowest exhibited the smallest response to the intervention and that higher performing students were the ones who benefit most.

This inequality increasing effect is magnified when taking into account both cross-school and crossstudent heterogeneity (Table 5 and 6 Columns 6-10). As before, teacher incentives only had significant impacts for schools with baseline average test scores above 50 percent —but even among those schools it was students at the higher quantiles for whom impacts are largest. In phase 2, for example, effect sizes increase systematically from 0.152 sd (=0.21-0.058) at the 10<sup>th</sup> percentile to 0.445 sd (=0.025+0.42) at the 90<sup>th</sup> percentile. These results are particularly concerning since they suggest that in initially low-performing schools the incentives may have had negative impacts on student learning outcomes. At the 10<sup>th</sup> percentile of the (conditional) test score distribution, we find that performance-based incentives were associated with statistically significantly lower test scores with effect sizes on the order of -0.06 to -0.07 sd. At other quantiles for these schools, impacts are also generally negative, but smaller and not statistically significantly different from zero.

The impact of the student incentives (either combined with teacher incentives in phase 1, or alone in phase 2) also had heterogeneous impacts across the (conditional) test score distribution—although the patterns are slightly more mixed. As mentioned above, when combined with teacher incentives (phase 1), 14
the size of the impacts increases from 0.1 sd at the 10<sup>th</sup> percentile to 0.21 sd at the 90<sup>th</sup> percentile (Table 5 Columns 1-5). This seems to be coming from a combination of impacts at lower percentiles in initially poorly performing schools and impacts at higher percentiles in initially higher performing schools (Table 5 Columns 6-10). In phase 2, the results are much more consistent and suggest that student incentives alone only had an impact in initially higher performing schools—and that this impact did not vary much across the (conditional) test score distribution; impacts are all between 0.21-0.25 sd (Table 6 Columns 6-10).

# **3.4 Sustainability of learning impacts**

The results from the treatment arm in which teacher incentives were withdrawn suggest that program impacts persisted even after the intervention ended. In that particular case it would appear that the additional effort induced by the incentives in phase 1 may have been sustained by the teachers beyond the period for which they were eligible for incentives. In this section we focus on the question of whether students of incentivized teachers, whose learning increased while their teacher was benefitting from the program, were able to sustain that boost to learning as they moved on to a higher grade and another teacher.

Table 7 reports the results for all students (Table 7 Columns 1 and 2) and for the sample restricted to those students who were actually attending the same school in the prior school year and would therefore have been a part of the incentive program if they were in a treated school (Table 7 Columns 3 and 4). The results confirm that the impacts we observed at the end of the year of program exposure were sustained through to the subsequent year. Average impacts of teacher incentives are statistically significant with an effect size of 0.12 sd (when excluding new students). The effects are concentrated among students who were attending schools that performed well at baseline, where the effect size is 0.34 sd (=0.372-0.030). On average we detect no impacts of student incentives, but allowing for heterogeneity reveals a marginally significant negative impact for students in initially low-performing schools (effect size of -0.10 sd) and a positive impact for those in initially high-performing schools (effect size of 0.25 sd (=0.351-0.104)).

# **3.5 Pathways to learning impacts**

The last step in our empirical analysis is to explore pathways through which effects might be generated, focusing on how teachers (and students) adjusted their behaviors to respond to the performance-based incentives. We do this along three dimensions: teacher attendance, teacher classroom behavior, and student attendance. In general, the results do not suggest that incentives operated through these channels.

Consistent with the results from a comparable intervention (Muralidharan and Sundararaman 2011a) we find that teachers did not increase attendance in response to incentives (Table 8). If anything, the results point to negative impacts of teacher incentives on teacher presence, although the sizes of these effects are 15
extremely small (e.g. a reduction of on the order of 4 percentage points compared to a base of about 80 percent). Across domains of teacher in-class behavior, we find significant effects of teacher incentives only in phase 2 in the assignment and review of homework domain. In particular, teachers assigned to the incentive arm have a 0.23 sd larger index on average relative to teachers in the control group (Table 9a Column 8). The point estimate in phase 1 is a modest positive but insignificant effect. Testing for heterogeneity in these effects (Table 9b Column 8) suggests more homework in initially high performing schools, but the point estimates are imprecise. We are unable to detect any statistically significant impacts on the other three classroom behaviors we measure—teacher-student interactions, use of materials, and use of board for lessons and examples (Table 9a and 9b columns 1-3, 5-7). While effect sizes are relatively large, for example on the order of 0.3 to 0.6 for teacher incentive impacts in initially high-performing schools in phase 1 (Table 9b Columns 1-3), these are estimated very imprecisely.

Finally, student attendance does not systematically change as a result of incentives (Table 10). We detect small increases as a response to teacher incentives in phase 1 (on the order of 6 to 8 percentage points compared to a base of 72-75 percent) but these are not consistent with the learning impacts (for example, these impacts are not larger for the initially high-performing schools) and are therefore not a likely pathway for those impacts.

# **4. Conclusion**

This paper examines the impacts of relatively low-cost performance-based incentives to teachers and students in Tanzanian secondary schools. By exploiting a two-phased randomized control trial we estimate average and heterogeneous impacts of incentives. In addition, we examine the effectiveness of incentivizing students (both along with teachers, as well as students only), establish the impact of withdrawing incentives, study the sustainability of those impacts, and explore potential pathways.

We find that the teacher incentives led to modest gains in student learning. Contrary to fears expressed by some policy makers, withdrawal of the performance-based incentives did not lead to discouragement effects. We document that when there were impacts, these were sustained beyond the intervention period. We do not find any evidence that teachers or students responded to incentives by increasing their attendance. Other than increasing homework assignment in phase 2, we find no evidence of other pedagogical behavioral responses across both phases. Finally, incentives aimed only at students did not lead to any average impacts, while those aimed at teachers or teachers and students together were more effective at increasing learning.

16
We document substantial heterogeneity in impacts which is concerning since the effect is to exacerbate learning inequalities. When there were positive impacts on learning, these were concentrated in schools that were initially high performing. Moreover, we find that impacts were largest for students at the top end of the (conditional) test score distribution. At the same time we find that, in initially lower-performing schools, students, particularly those at the bottom end of the (conditional) test score distribution, may have been negatively impacted by the incentives. So, while performance-based teacher and student incentives may have led to modest average positive impacts on learning, this came at the cost of widening gaps between lower- and higher-performing students and schools and potentially even negatively impacting some students. Note that we cannot identify the mechanism by which inequality is exacerbated. For example, it could be because incentivized teachers concentrate their additional effort on initially higher performing students. Or it could be the case that they provide equal additional effort across all students, but only initially high-performing students benefit from that effort (perhaps because of a curriculum mismatch between the material taught and the distribution of student skills as discussed in Kaffenberger and Pritchett 2020). We cannot distinguish between these mechanisms.

Our findings have both research and policy implications. On the research side they suggest that we need to better understand how curriculum difficulty and the incentives teachers face to prioritize curriculum completion versus student mastery condition teachers' efforts across the student ability distribution. On the policy side, they suggest that future interventions that deploy performance-based incentives—in Tanzania and elsewhere—should take care to mitigate potential inequality-increasing effects.

17