metadata: authors: 0: author: Barrera-Osorio,Felipe
1: author: Cilliers,Jacobus
2: author: Cloutier,Marie-Helene
3: author: Filmer,Deon P.
count: Guinea
docna: 0: docna: Heterogenous Teacher Effects of Two
            Incentive Schemes : Evidence from a Low-Income Country
docty: Policy Research Working Paper
subtopic: Effective Schools and Teachers,Educational Institutions & Facilities,Communicable Diseases,Gender and Development,Educational Sciences,Public Health Promotion
teratopic: Education,Health, Nutrition and Population,Gender
abstracts: cdata!: This paper reports on a randomized
            evaluation of two teacher incentive programs, which were
            conducted in a nationally representative sample of 420
            public primary schools in Guinea. In 140 schools,
            high-performing teachers were rewarded in-kind, with the
            value of goods increasing with level of performance. In
            another 140 schools, high-performing teachers received a
            certificate and public recognition from the government.
            After one year, the in-kind program improved learning by
            0.24 standard deviations, while the recognition treatment
            had a smaller and statistically insignificant impact. After
            two years, the effect from the in-kind program was smaller
            (0.16 standard deviations) and not significant; the paper
            provides suggestive evidence that the reduction may be due
            to the onset of an Ebola outbreak. The effects of the
            recognition program remained small and insignificant. The
            effects differed by teacher gender: for female teachers,
            both programs were equally effective, while for male
            teachers, only the in-kind program led to statistically
            significant effects.
geo_reg_key: 517191,1927320
repnme: repnme: Heterogenous Teacher Effects of Two
            Incentive Schemes : Evidence from a Low-Income Country
display_title: Heterogenous Teacher Effects of Two
            Incentive Schemes : Evidence from a Low-Income Country
listing_relative_url: /research/2021/05/33059872/heterogenous-teacher-effects-two-incentive-schemes-evidence-low-income-country
topicv3: Teachers Management m1326119 1523,Education m1327048 470,Pandemics m1327520 2167,Gender m1327621 615,Ebola m1504702 2663
publishtoextweb_dt: 2021-05-07T00:00:00Z
dois: 10.1596/1813-9450-9652
historic_topic: Education,Health, Nutrition and Population,Gender
origu: Development Research Group (DECRG)
no_of_pages: 39
author_keys: 0: author_key: 1060969
1: author_key: 97345
model_extractions: Policy Research Working Paper 9652 Heterogenous Teacher Effects of Two Incentive Schemes Evidence from a Low-Income Country _Felipe Barrera-Osorio Jacobus Cilliers Marie-Hélène Cloutier Deon Filmer_ Development Economics Development Research Group & Education Global Practice May 2021
Policy Research Working Paper 9652

# **Abstract**

This paper reports on a randomized evaluation of two teacher incentive programs, which were conducted in a nationally representative sample of 420 public primary schools in Guinea. In 140 schools, high-performing teachers were rewarded in-kind, with the value of goods increasing with level of performance. In another 140 schools, high-performing teachers received a certificate and public recognition from the government. After one year, the in-kind program improved learning by 0.24 standard deviations, while the recognition treatment had a smaller and statistically insignificant impact. After two years, the effect from the in-kind program was smaller (0.16 standard deviations) and not significant; the paper provides suggestive evidence that the reduction may be due to the onset of an Ebola outbreak. The effects of the recognition program remained small and insignificant. The effects differed by teacher gender: for female teachers, both programs were equally effective, while for male teachers, only the in-kind program led to statistically significant effects.

This paper is a product of the Development Research Group, Development Economics and the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http:// www.worldbank.org/prwp. The authors may be contacted at ejc93@georgetown.edu, felipe.barrera.-.osorio@vanderbilt. edu; mcloutier@worldbank.org; or dfilmer@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
Heterogenous Teacher Effects of Two Incentive Schemes: Evidence from a Low-Income Country<sup>_∗_</sup> Felipe Barrera-Osorio<sup>_†_</sup> , Jacobus Cilliers<sup>_‡_</sup> , Marie-H ́el`ene Cloutier<sup>_§_</sup> Deon Filmer<sup>_§_</sup> _Keywords:_ Student achievement; Teacher incentives; Learning outcomes; Recognition rewards _JEL Codes_ : I21; I28; J33; J45; O15

> _∗_ Acknowledgements: We thank a number of colleagues and institutions that have assisted at various stages of the evaluation. Nathalie Lahire, Karine Angles, and Assane Dieng who played an instrumental in support the preparation and implementation of the evaluation. Adama Tiendrebeogo who provide both essential in-country field work coordination and research assistance throughout. Diwakar Kishore, Youn Park, and Ana Alvarez who provided critical research assistance to the project. Faya Fran ̧cois Bourouno and Abdoul Aziz Diallo who supported in-country field work and supervision of data collection. This work would not have been possible without the impressive commitment from the Ministry of Pre-University and Civic Education (MEPU-EC), especially the teams of the Inspection G ́en ́erale de l’Education (IGE) and the<sup> ́</sup> Cellule Nationale d’Evaluation<sup> ́</sup> du Syst`eme Educatif<sup> ́</sup> (CNESE). We acknowledge funding from the Strategic Impact Evaluation Fund (SIEF) and the Research Support Budget (RSB) of the Development Economics unit (DEC). All findings and interpretations in this paper are those of the authors, and do not necessarily represent the views of their respective institutions or the Government of Guinea.

> _†_ Vanderbilt University

> _‡_ Georgetown University. ejc93@georgetown.edu

> _§_ World Bank
# **1 Introduction**

Teacher incentive programs are a promising policy tool to improve student learning outcomes in developing countries. This hope stems from the fact that most developing countries currently face a “learning crisis” (World Bank, 2018), and there is evidence of limited teacher accountability (Mbiti, 2016). In fact, multiple studies have demonstrated the success of teacher pay-for-performance schemes in improving student learning outcomes in developing countries. Yet few governments have adopted them, either due to political reasons —teacher trade unions are often opposed to them— or ideological resistance to the idea. In contrast, governments frequently implement policies or have informal systems in place that reward high-performing civil servants through public recognition, such as award ceremonies or certificates. However there is little evidence on the effectiveness of these non-pecuniary incentive schemes. Such schemes activate different sources of motivation —reputational and intrinsic, rather than financial— and so could have very different impacts, depending on the teacher (B ́enabou and Tirole, 2006).

This paper presents experimental evidence that allows us to investigate the effects of both types of approaches. We have two main research questions. First, whether a performancebased reward can result in better student learning outcomes, when implemented at a national level by government in a low-income country. Second, whether there is a difference in the effect of in-kind rewards versus rewards that only provide “recognition” for performance. We further explore whether there is heterogeneity across teachers in the effects of these reward schemes.

We analyze a nationally representative sample of 420 public primary schools in Guinea, situated in West Africa. These schools were assigned to one of three treatment arms. In 140 schools, relatively high-performing teachers were rewarded in-kind, with the value of goods increasing with level of performance (rice sacks, radios, phones, televisions, and generators). In another 140 schools, relatively high-performing teachers received a certificate and public recognition from government. The remaining 140 schools were assigned to the control group. Performance was calculated using a weighted average of (i) the annual gain in the average score of the teacher’s students on standardized tests; and (ii) the teacher’s annual improvement in the quality of his/her lessons preparation and delivery, as measured during inspection visits.

We conducted three rounds of data collection: at baseline, prior to the start of the program (May 2012), at midline after the first year of implementation (May 2013), and at endline after two years (May 2014). During these visits, fieldworkers conducted school principal and teacher surveys, as well as student assessments in mathematics and French 2
literacy. In addition, school inspectors visited schools to assess the quality of teaching and lesson preparation. Steps were taken to assure independence in data collection and minimize risk of collusion between and teachers.

We find that at midline, after one year of program implementation, there is a sizable and statistically significant impact on test scores (about 0.24 standard deviations, sd) for the in-kind treatment arm, and a much smaller and statistically insignificant impact for the recognition treatment arm (about 0.13 sd). At endline, after two years of program implementation, the impact of the in-kind rewards had fallen by about 30 percent in magnitude (to 0.16 sd) and is no longer statistically significantly different from zero. The impact of recognition rewards remains smaller (0.09 sd) and statistically insignificant . We also find that at midline the impacts are larger for girls than boys. We find no evidence of strategic behavior on the number of students who were assessed in each year, in average enrollment rates, or in changes in enrollment. We provide suggestive evidence that the reduction in impacts at endline is linked to the growing Ebola epidemic that Guinea experienced at this time.

Effects differ by teacher gender. The impact of in-kind rewards are very similar across genders, 0.20 sd and 0.18 for female and male teachers respectively. In contrast, the effects of the recognition rewards is 0.23 sd for female teachers, while it is 0.01 sd (and insignificantly different from zero) for male teachers. The findings further suggest that the positive effects of either scheme for female teachers benefit both female and male students. However, the positive effects of the in-kind treatment for male teachers benefit only female students. This is consistent with targeted action since female students have lower test scores at baseline.

Finally, data from the school inspections suggest an improvement in teaching preparation and practice. Teachers in the in-kind treatment arm made better use of resources in the classroom, were better prepared for the class, and implemented better teaching practices. Consistent with the results on student learning, the impacts on teaching practice are no longer statistically significant at endline.

Our paper adds to a growing literature with experimental evidence from teacher pay-forperformance interventions in low- and middle-income countries. This literature documents a range of findings: some studies find no impacts (Barrera-Osorio and Raju, 2017); others document positive impacts (Behrman et al., 2015; Cilliers et al., 2018; Filmer et al., 2020; Leaver et al., 2020; Loyalka et al., 2019; Muralidharan and Sundararaman, 2011); and others have shown impacts only in the presence of complementary resources (Gilligan et al., 2018; Mbiti et al., 2019). Last, one study finds impacts on the incentivized test, but no evidence of impacts on any other learning outcomes (Glewwe et al., 2010). In contrast, null and negative impacts have been typically been found in experiments in high-income settings 3
(Fryer, 2013; Fryer Jr, 2011). Given these mixed results, it is clear that context and the details of design matter greatly for determining the impacts of performance-based rewards (Bruns et al., 2011). In some contexts teachers may lack the skills and knowledge to respond effectively to incentives, even if they wished to.

Our contribution is both to the design and to the discussion of context specificity of these program. First, we document the contrast in the impacts between in-kind and recognition rewards, and explore heterogeneity in how these effects are mediated by teacher and student gender. To the best of our knowledge, this is the first paper with a clear identification strategy to look at the contrasts between financial and recognition rewards for government civil servants. A similar study experimentally compared financial incentives with providing recognition for pro-social behavior, although the agents were private-sector hairdressers (Ashraf et al., 2014). Second, we document impacts on learning outcomes even in an environment where teacher skills are relatively low.

# **2 Intervention, Sample, and Evaluation Design**

Guinea, located on the west coast of Africa, is one of the poorest countries in the world. It is ranked 174/189 on the Human Development Index and, as of 2012, 36 percent of the population lived in extreme poverty.<sup>1,2</sup> School enrollment at all education levels have been increasing in recent years, but serious challenges remain both in access and quality. For example, an assessment administered in 2017 revealed that only 25 percent of grade two and three children could read a simple text,<sup>3</sup> and according to the Human Capital Index a child born today can expect to complete 7 years of schooling, but adjusting for quality suggests that amounts to only 4.5 years of high-quality schooling (World Bank, 2018).

These apparent problems induced a newly elected government in 2011 to seek innovative approaches in personnel and human resource management reforms. Within this context, the Ministry of Pre-University and Civic Education (MEPU-EC) initiated a donor-funded individual performance-based reward program for teachers. The program started in the 2012-2013 academic year and was implemented relatively well (see discussion below), but faced serious challenges in its last year of implementation (2013-14 school year) due to the outbreak of Ebola in December 2013, and especially as the epidemic spread in 2014.<sup>4</sup>

> 1 `http://hdr.undp.org/en/data`

> 2 `https://data.worldbank.org/country/guinea`

> 3 `https://resenguinee.org/`

> 4Source: `https://www.cdc.gov/vhf/ebola/history/2014-2016-outbreak` Accessed September 5th, 2020

4
# **2.1 Performance-Based Reward Program**

The program was designed in collaboration with government and key stakeholders like teacher unions. While it used principles derived from optimal design, the approach emphasized simplicity in the pursuit of broad understanding. The program targeted grade 3 and 4 teachers, and lasted for two academic years (academic years 2012-13 and 2013-14). At these grades, teachers are assigned to a specific group of students and teach all subjects for that group.<sup>5</sup> Rewards took the form of either goods (in-kind) or recognition, depending on the treatment arm to which the teacher’s school was assigned. The value of the reward was determined on an absolute scale, without relative performance comparisons to other teachers.<sup>6</sup> The performance indicator used in both programs was a weighted sum of improvement in learning outcomes and improvements in observed teacher inputs. The indicator measuring learning outcomes was the annual gain (in percentage points) in the average score of the teacher’s students on standardized tests in Math and French. The input indicator was the teacher’s annual improvement in the quality of his/her lessons preparation and delivery, defined as the annual gain (in percentage points) in the inspection scores given to the teacher for two lessons —in in French and one in Math— during inspection visits.<sup>7</sup> The sub-indicators were aggregated using a weight of 0.7 for improvement in learning outcomes and 0.3 for improvement in teacher inputs. In the second year a third indicator, the student attendance rate on the day of the test, was introduced to disincentivize teachers from encouraging some students to stay home on the test day. Weights were then modified to be 0.6, 0.3 and 0.1. Data on these performance sub-indicators were collected through announced visits. Specific effort was made in the design and administration of the assessments and inspections to minimize cheating and collusion. First, the research team used a randomized booklet design, where students in a school were randomly assigned to different test booklets that contained different test items covering the same subject subdomain and some overlapping items. One comparable learning score was constructed for each student _ex post_ using Item Response Theory.<sup>8</sup> Second, test booklets were kept secure prior to administration and answer keys were never distributed, either before or after test administration. Third, tests were administered in the schools by external agents recruited by the education ministry. Fourth,

> 5Except for koranic lessons for which a special teacher is assigned.

> 6Section A.1 provides more details on the types of reward, actual receipt of rewards (Tables A.1 and A.2), implementation of the award ceremonies, and communication strategy to inform schools and teachers about the program.

> 7Both sub-indicators took on a value of zero if the teacher’s performance worsened.

> 8Items from a large pool of potential test items were piloted with students in a set of schools outside of the study sample. Statistical analysis of the piloting data was carried-out to identify items that can be combined into booklets to give accurate and comparable assessment results.

5
administration conditions, timing, resources and environment were kept as consistent as possible between schools. Fifth, the inspections and classroom observations were conducted by regular inspectors from the ministry, but they were assigned to a region different from their regular assignment to increase objectivity and reduce the risk of collusion between the teachers and inspector.

The program roll out was accompanied by a strong communication campaign (conducted in December 2012) to teachers and principals. This communication campaign was implemented by a specialized consulting firm using pamphlets, posters, and meetings with communication agents. Understanding of the program was also reinforced during the award ceremonies and disbursement of gifts at the end of the first year (December 2013-January 2014). In the second year, ceremony and reward distribution were initially delayed because of the Ebola crisis (which had led to a ban on people gathering in groups), and were ultimately canceled altogether. See Figure A.1 for the timeline of implementation, data collection, and outbreak of Ebola.

Overall, teachers’ awareness of the program was high, but knowledge about the specific aspects of program design was weaker (Table A.3). Almost all teachers, whether in the incentive treatment or in the control group, reported being part of the program. We suspect that this is due to their (incorrect) interpretation of the regular visit by inspector and teams from ministry for the student test and interviews. It is thus a possibility that we are underestimating effect sizes if teachers in the control schools also changed their behavior due to incorrect beliefs about their participation in the program. In general, knowledge of key aspects of the program were high, but not perfect. For example, at midline about 75 percent of teachers in the treatment groups—but not all—believed that the program targeted grade 3 and 4 teachers. At midline 91 percent of teachers in the in-kind arm believed that the award had an in-kind component, and 62 percent of teachers in the recognition arm believed that the program included a certificate. Teachers were also relatively well-informed about the inputs to the award function.

# **2.2 Sample and Experimental Design**

The evaluation sample consists of 420 randomly selected primary schools, stratifying by regions and zones (rural-urban), from the population of public French (non-Arabic) speaking schools found in the Education Management Information System (EMIS) database and using the number of teachers in targeted grades to define the selection probability of a given school.<sup>9</sup>

> 9The sampling frame was limited to public schools because they are the schools under direct authority of the MEPU-EC.

6
The sample is therefore nationally representative for grade 3 and 4 teachers in public Frenchspeaking schools in Guinea.

Out of this sample, 140 schools were randomly assigned to each of the two treatment arms —recognition or in-kind performance rewards— and 140 schools were randomly assigned to the control. Random assignment was carried out publicly in the context of a program launching workshop, after the baseline data collection. Names of schools were put in bowls (per strata) and picked (blindly) by young children to be assigned to one of the groups. The public nature of the randomization process aimed at maximizing trust in the transparent and impartial nature of the allocation process.

# **2.3 Data**

Data were collected on schools, teachers, and students using the following survey instruments: (i) a questionnaire administered to the school’s principal, (ii) a teacher questionnaire administered to targeted teachers (grade 3 and 4), (iii) a Math test and French test (with different parallel booklets) administered to students in the targeted teachers’ classrooms, and (iv) an inspection bulletin administered to targeted teachers in the context of two lessons, one in French and one in Math. Table A.4 presents a more detailed description of the various instruments.

All grade 2 and 3 students in the sampled schools were assessed at baseline (May 2012), and all students in grades 2 to 4 were assessed at midline (May 2013). Because of budget constraints, a random sample of up to 25 students were assessed from each class in grades 3 and 4 at endline (May 2014). The surveys reached a varying number of schools across the different rounds (Table A.5). For example, we have student assessment data for 408, 417, and 391 schools for baseline, midline, and endline rounds of data collection, respectively. It is unclear the source of the variation in number of schools; however, it is likely that, at least for the endline, the outbreak of Ebola might have been a contributing factor.

Most data collection activities were carried out using the existing government systems (no external survey firm was used) and took place simultaneously in each region of the country. Student and teacher tests, as well as principal, teacher and student questionnaires were administered by decentralized government staff selected on a competitive basis and trained as enumerators and supervisors. Quality of the data collected through the above mechanisms was validated through spot check visits from field coordination teams.

The data collection effort was set up as a panel of schools, not of teachers or students. Some teachers —those still teaching in the surveyed grade— appeared in multiple rounds of the data collection; other teachers were new and others left the grade of the school. As such, 7
we do not have a panel structure for teachers.

Due to budget limitations, the analysis of impacts and the calculation of rewards use the same tests. Given that the program is attached to the same test, there may be problems of degradation and inflation of the metric (Koretz, 2002; Neal, 2013). Nevertheless, we use a measure of learning outcomes that is different from the metric used to calculate the incentive payments. While the latter uses an IRT model to account for differences between grades, our estimation of impacts use a grade fixed-effects specification. In addition, as discussed above, the measures taken to minimize cheating mitigate this concern. Furthermore, it is unlikely that teachers were merely “teaching to the test” in the first year of implementation, since teachers had no experience with the tests.

At the time of baseline data collection, teachers’ mean monthly salaries were USD118, 73 percent of teachers felt that their salary was insufficient, 27 percent of teachers felt that they received insufficient recognition for their work, 67 percent were female, and 70 percent mentioned that they would choose teaching as a profession if they could choose over (Table A.6). There were also large disparities by gender. Female teachers reported to have larger classes, earn less, and were less satisfied with their earnings, but were also 10 percentage points _more_ likely to state that they would choose teaching again as a profession, if they had the chance to choose over.

In addition to the primary data collection, we also constructed a data set containing the total number of reported deaths due to Ebola for each prefecture by May 2014, the time of endline data collection. We combined the weekly reports on the total number of new cases by prefecture, provided by the WHO. The distribution of known deaths due to Ebola by May 2014 across prefectures is highly varied with the majority of cases were in one prefecture (168 cases in Gueckedou), and just over 65% of the prefectures had no known cases (Figure A.2).

We perform a range of tests to demonstrate the internal validity of the study.<sup>10</sup> The sample is balanced for a range of student and school-level characteristics, including student baseline learning, number of students assessed, and exposure to Ebola (Table A.7). Furthermore, school-level attrition in student assessment data is balanced (Table A.8), and the sample remains balanced if restricted to schools for whom we have midline and endline data (Tables A.9 and A.10, respectively). Finally, there is no _reduction_ in the number of students assessed at midline and endline in the respective treatment groups, suggesting that teachers did not strategically keep under-performing students from attending school on the day of assessment to inflate their scores (Tables A.11 and A.13).

> 10Section A.2 provides a more detailed discussion.

8
## **2.4 Empirical Strategy**

Our main estimating equation is the following: where _Yigas_ is learning outcome for student _i_ , grade _g_ , assessment _a_ (Math or French language assessment), and school _s_ . _ρb_ refers to strata fixed effects.<sup>11</sup> _Xigs_ is a vector of school-level and student-level controls: the head-teacher’s and students’ age and gender, and the school’s average baseline performance for grades 2 and 3, respectively.<sup>12</sup> _λg_ and _γa_ are grade and assessment For all specifications, we cluster standard errors at the school level. We run two separate regressions for midline and endline observations. Observations are weighted by the inverse of the number of students assessed in the school. This model allows us to directly test the impact of each type of reward approach (i.e. the coefficients _β_ 1 and _β_ 2) as well as test whether the impacts are different for the two approaches (i.e. the test that _β_ 1 = _β_ 2).

For the heterogeneity analysis by teacher gender, we estimate Equation 1 separated for male teachers and female teachers.

When testing whether treatment effects vary by student gender, we estimate the following: This model allows us to test the impact of each type of reward approach for girls (i.e. the coefficients _α_ 1 and _α_ 2), for boys ( _α_ 1 + _α_ 3 and _α_ 2 + _α_ 4), and the difference in effect size between boys and girls ( _α_ 3 and _α_ 4). Moreover, we can test whether the impacts for the two treatments are the same, again separately for girls ( _α_ 1 = _α_ 2) and boys ( _α_ 1 + _α_ 3 = _α_ 2 + _α_ 4).

# **3 Results**

## **3.1 Student Test Scores**

Table 1 presents our main results. At midline, after one year of program implementation, the in-kind and recognition reward programs improved student learning by 0.24 and 0.13

> 11The assignment across groups is balanced within each stratum.

> 12If a school does not have baseline data for a specific grade, we assign it to the mean value in the control and create a dummy variable equal to one if data is missing.

9
standard deviations, respectively. We cannot reject the null of no impact for the recognition program (p-value= 0 _._ 205), nor the null for equal effects across arms (p-value 0.195; last row of the table). There is evidence of decay in the effect of the program: at endline, after two years of program implementation, the impact of the in-kind and recognition reward programs were smaller, 0.16 and 0.09 sd respectively, and statistically insignificant.

Table 1 also unpacks effects by grade (Columns (3) to (6)) and by subject (Columns (7) to (10)). The impacts are consistently larger for grade 4 students, relative to grade 3 students, for both treatments and both rounds of data collection.<sup>13</sup> The effects of the inkind arm in French and math were similar and positive at midline; the analogous effects at endline are still positive, but only statistically significant for French. These results by grade and subject consistently show positive and higher point estimates of the in-kind treatment than the recognition arm, despite the fact that we cannot statistically rule out equality at conventional levels of significance.

This program is very cost-effective (Table A.12). Assuming that a random sample of 25 students per school are assessed in each round, and that the cost of the midline ceremonies and goods distribution would have carried over to the endline had they occurred, we estimate that the in-kind and recognition programs cost approximately $11.75 and $8.29 per student per year respectively (including costs of data collection, award ceremonies, and value of goods awarded). The in-kind program is thus more cost-effective with a 2 sd increase in learning for each $100 spent per student, relative to 1.48 sd/$100 for recognition. We note that the biggest cost-driver is the cost of assessing students, which explains why the recognition rewards program was only 29 percent less expensive than in-kind rewards.

# **3.2 Teacher Heterogeneity**

We examine two sources of teacher heterogeneity: teacher gender, and teacher “identity” (defined below). Our analysis is constrained by the fact that we cannot link teachers with students, although we can match at a school-grade level. The data is therefore restricted to school grades where there is only one teacher, or all the teachers in that grade share the same characteristics.

Table 2 examines this heterogeneity. We have 86,840 observations for the whole sample (Table 1, Column 1) and 58,949 observations for the sample in which we can cleanly establish the gender of the teacher. Column (1) replicates the overall result on test scores at midline for the sample with teacher gender. It is reassuring that the point estimates are very similar to the ones in Table 1, Column 1. Columns (2) and (3) presents the estimates disaggregated 13In line with fact that the program only targeted teachers in grades 3 and 4, we find no impact on learning for grade 2 students at midline. These results are available upon request.

10
by teacher gender. The effect of in-kind incentives is very similar for both female and male teachers (point estimates of 0.19 and 0.19 sd respectively). In contrast, the effect of the recognition incentive varies with teacher’s gender. The point estimate of the recognition treatment for female teachers is 0.23 sd (and is statistically significant at conventional levels), while the effect for male teachers is 0.004 sd. We cannot reject the null hypothesis of equality between the impacts of the in-kind and recognition treatments for male teachers, due to weak statistical power.

In short, there is suggestive evidence that the impacts of these performance incentives vary with the gender of the teacher: the effects on test scores of students from female teachers are higher for _both_ treatments, while the effects from male teachers are driven only for the in-kind treatment.

We posit two potential reasons why female teachers are more responsive to the recognition program than male teachers that we can empirically investigate. First, it is plausible that a public recognition of one’s professional competence is more valuable to individuals whose personal identity is linked to that profession. Second individuals who feel underrecognized might respond stronger to opportunities to increase the recognition of their work. It is possible that female teachers identify more strongly as teachers, and also receive lower recognition for their work overall.

We find some suggestive evidence for the former hypothesis, but not the latter. As a coarse measure of “identity”, we asked teachers what profession they would choose if they could choose again. At baseline, female teachers are 10 percentage points more likely to indicate that they would choose teaching as a profession again: 77 vs 67 percent. Columns (3) to (6) in Table 2 shows the same pattern of heterogeneous effects based on this question. Teachers who respond that they would choose teaching as a profession respond equally well to both the in-kind and recognition reward programs. However, the impact of the recognition arm for teachers who would _not_ choose the profession again is much smaller —0.007 sd— and not statistically significant. Again, we cannot reject the null hypothesis of equality between the impacts of the in-kind and recognition treatments for teachers who would not choose the profession, due to limited power. In contrast, we find no evidence that female teachers are more likely to be unsatisfied at baseline with the level of recognition they receive for their work. Moreover, there is no evidence that teachers who are dissatisfied with the recognition they receive for their work responded more to the recognition program.<sup>14</sup> We further disaggregate the results by the gender of the student, by estimating Equation 2. The first column in Table 3 shows that in both programs girls benefited more than boys. The difference in effect size between boys and girls is largest for the recognition arm (0.12

> 14Results available upon request.

11
sd), and statistically significant at the 5 percent level. As a result, both programs had a positive and statistically significant impact on learning for girls at midline. Note that boys in the control group out-perform girls by 0.18 sd on average. This means that the programs also succeed at reducing the difference in performance between boys and girls, by roughly a third and two thirds in the in-kind and recognition arms respectively.

In columns (3) and (4) we report the same results as above, but for female and male teachers separately. Again, the comparison between Columns (1) and (2) is important since it confirms that the restricted sample for which we can ascertain a teacher’s gender does not drive the results. The effect for a female teacher on female students is 0.21 sd for the in-kind treatment, and 0.24 sd for the recognition treatment; with both being statistically significantly different from zero at conventional levels. The effects of a female teacher on male students is very similar: the interactions between treatment arm and the boy indicator are small and non-significant. In contrast, there are differential effects for male teachers (Column 4). In this case, the effect for a male teacher on a female student in the in-kind treatment is 0.27 sd, while the effect on male students is 0.12 (0.27-0.15) sd; we note that the coefficient on the interaction term, -0.15, is not statistically significant. The effect for a male teacher on a female student from the recognition treatment is 0.13, which is not statistically significant different from zero, and the effect on male student is -0.08 (0.13-0.21) sd which is negative but not statistically significantly different from zero.

In sum, these estimates present an intriguing perspective of the effects of teacher incentive programs. On one hand, there is supporting evidence that female teachers respond to both pecuniary and non-pecuniary incentives, and do so in ways that benefit all students, male and female. On the other hand, the estimates indicate that male teachers respond mainly to the pecuniary incentive, and do so in a way that benefits only female students. This behavior is compatible with them taking a strategic approach, since female students of male teachers tend to have markedly lower counterfactual test scores than male students.

# **3.3 Inspectors’ Classroom Observations**

We further investigate the ways through which teachers may have responded to incentives by using the data on teacher practices collected through the inspector classroom observations. The inspector assigned each teacher scores on two indicators of classroom quality and five indicators of teaching quality. We standardized each score to have a control group mean of zero and standard deviation of one, and then took the mean of these standardized scores to create an overall index of classroom and teaching quality, respectively. Table 4 reports the effects of the program on these indicators at midline. The specification is similar to that for 12
Equation 1, but with the outcome variables at the level of the teacher.

Overall, we see positive effects from the in-kind treatment, with effect sizes of 0.14 sd on the overall index of classroom quality, and 0.11 sd on the overall index of teaching quality (with both being statistically significantly different from zero). In contrast, the effects for the recognition arm are not statistically significantly different from zero, and both point estimates are lower than those for the in-kind treatment. Breaking out the sub-components of these overall indices yields systematically positive effects from the in-kind treatment, but varied impacts (both in size and direction) of the recognition treatment.

We take these results with some caution: the evaluation of the inspector is an important input for the provision of the rewards. In the case of the in-kind treatment, there may have been a perception that there was more at stake (since the reward is material), and we cannot rule out that the positive results may be driven by collusion between the inspector and teacher. We do not have evidence of this kind of behavior (or lack thereof). Mitigating this concern is the fact that we actually observe student effects from the in-kind treatment: If inspector observations were driven purely by collusion, we would have expected to find no impacts on student outcomes.

# **3.4 Exposure to Ebola**

We present evidence that the outbreak of Ebola might have been a contributing factor for the reduction in treatment effect sizes at endline, relative to midline. Table 5 presents the results from estimating a model that relates student test scores as a function of the treatment arms, an indicator of Ebola, and the interaction of Ebola and treatments. As a placebo test, column (1) shows that the magnitude of the treatment effects did not vary by exposure to Ebola at midline, prior to the actual outbreak of the disease. Column (2) shows that at endline the treatment effects for both interventions are smaller in magnitude in prefectures with higher exposure to Ebola. For both interventions, the effect sizes decrease by 0.006 standard deviations for every additional death from Ebola. The treatment effects in the prefectures not exposed to Ebola are very similar in magnitude to the average treatment effects at midline. This suggests that all the reduction in effect sizes between midline and endline could be attributed to the Ebola outbreak.<sup>15</sup>

> 15These results are robust to different transformations of the Ebola variable (e.g. hyperbolic sine, Table A.14), and to the exclusion of five prefectures that had not reported Ebola cases by the time of endline data collection (Table A.15).

13
# **4 Discussion and Conclusion**

Teacher incentives have the potential to encourage teachers to exert more effort and ultimately lead to increase in student learning, but they could also lead to distortionary behaviors which could be detrimental for learning. The details of program design have been shown to matter for impacts, especially if they interact with teachers’ intrinsic motivations differently. In the context of primary schools in Guinea, this impact evaluation shows that teacher incentives led to improvements in student learning outcomes after one year of program implementation —with in-kind rewards having impacts that are roughly twice as large as those of recognition rewards. After two years of the program, the impacts are substantially muted —they roughly halved in size— and no longer generally statistically significantly different from zero. This reduction in effect sizes was likely driven by the Ebola epidemic that was emerging close to the two-year mark of the program. Analysis of enrollment patterns across years and across cohorts does not suggest that there were any perverse enrollment impacts of the program. Data from the school inspections suggest an improvement in teaching preparation and practice: teachers made better use of resources in the classroom, were better prepared for the class, and implemented better teaching practices. Moreover, the results from lesson observations mirror the results on student learning.

Male and female teachers respond differently to the different types of incentives. While there are many potential explanations for this result, we have suggestive evidence that the different types of incentives may be activating different underlying sources of motivation. In particular, the degree to which a teacher identifies themselves with the profession—potentially signaling a greater degree of intrinsic motivation—might be related to their responsiveness to the recognition incentive.

These results have potential implications for policy. They suggest that teachers do indeed respond to performance incentives, and can do so in ways that improve student learning outcomes, even in a low-capacity environment. However, non-pecuniary rewards—which are often preferred by policy makers—might only be effective for some types of teachers.<sup>16</sup> This means that the underlying motivations of teachers will be important in determining the effectiveness of such interventions, and that policy makers should invest in understanding those motivations prior to implementation.

> 16This result complements Leaver et al. (2020) who show in Rwanda that there is a range in the degree to which teachers are intrinsically motivated, and that the less intrinsically motivated teachers are more responsive to pecuniary performance-related rewards.

14
# **5 Reference**

# **References**

- **Ashraf, Nava, Oriana Bandiera, and B Kelsey Jack** , “No margin, no mission? A field experiment on incentives for public service delivery,” _Journal of public economics_ , 2014, _120_ , 1–17.

- **Bank, World** , “The Human Capital Project,” _World Bank_ , 2018.

- **Barrera-Osorio, Felipe and Dhushyanth Raju** , “Teacher performance pay: Experimental evidence from Pakistan,” _Journal of Public Economics_ , 2017, _148_ , 75–91.

- **Behrman, Jere R, Susan W Parker, Petra E Todd, and Kenneth I Wolpin** , “Aligning learning incentives of students and teachers: Results from a social experiment in Mexican high schools,” _Journal of Political Economy_ , 2015, _123_ (2), 325–364.

- **B ́enabou, Roland and Jean Tirole** , “Incentives and prosocial behavior,” _American economic review_ , 2006, _96_ (5), 1652–1678.

- **Bruns, Barbara, Deon Filmer, and Harry Anthony Patrinos** , _Making schools work: New evidence on accountability reforms_ , The World Bank, 2011.

- **Cilliers, Jacobus, Ibrahim Kasirye, Clare Leaver, Pieter Serneels, and Andrew Zeitlin** , “Pay for locally monitored performance? A welfare analysis for teacher attendance in Ugandan primary schools,” _Journal of Public Economics_ , 2018, _167_ , 69–90.

- **Filmer, Deon, James Habyarimana, and Shwetlena Sabarwal** , “Teacher Performance-Based Incentives and Learning Inequality,” 2020.

- **Fryer, Roland G** , “Teacher incentives and student achievement: Evidence from New York City public schools,” _Journal of Labor Economics_ , 2013, _31_ (2), 373–407.

- **Gilligan, Daniel O, Naureen Karachiwalla, Ibrahim Kasirye, Adrienne M Lucas, and Derek Neal** , “Educator incentives and educational triage in rural primary schools,” Technical Report, National Bureau of Economic Research 2018.

- **Glewwe, Paul, Nauman Ilias, and Michael Kremer** , “Teacher incentives,” _American Economic Journal: Applied Economics_ , 2010, _2_ (3), 205–27.

- **Jr, Roland G Fryer** , “Financial incentives and student achievement: Evidence from randomized trials,” _The Quarterly Journal of Economics_ , 2011, _126_ (4), 1755–1798.

15
- **Koretz, Daniel M** , “Limitations in the use of achievement tests as measures of educators’ productivity,” _Journal of human resources_ , 2002, pp. 752–777.

- **Leaver, Clare, Owen Ozier, Pieter Serneels, and Andrew Zeitlin** , “Recruitment, Effort, and Retention Effects of Performance Contracts for Civil Servants,” 2020.

- **Loyalka, Prashant, Sean Sylvia, Chengfang Liu, James Chu, and Yaojiang Shi** , “Pay by design: Teacher performance pay design and the distribution of student achievement,” _Journal of Labor Economics_ , 2019, _37_ (3), 621–662.

- **Mbiti, Isaac, Karthik Muralidharan, Mauricio Romero, Youdi Schipper, Constantine Manda, and Rakesh Rajani** , “Inputs, incentives, and complementarities in education: Experimental evidence from Tanzania,” _The Quarterly Journal of Economics_ , 2019, _134_ (3), 1627–1673.

- **Mbiti, Isaac M** , “The need for accountability in education in developing countries,” _Journal of Economic Perspectives_ , 2016, _30_ (3), 109–32.

- **Muralidharan, Karthik and Venkatesh Sundararaman** , “Teacher performance pay: Experimental evidence from India,” _Journal of political Economy_ , 2011, _119_ (1), 39–77.

- **Neal, Derek** , “The consequences of using one assessment system to pursue two objectives,” _The Journal of Economic Education_ , 2013, _44_ (4), 339–352.

- **World Bank** , _World Development Report 2018; Learning to Realize Education’s Promise_ , Washington, DC: World Bank, 2018.

16
Table 1: Impacts on learning— by round, grade and subject

||Full s|ample|Gra|de 3|Grad|e 4|Fre|nch|Ma|th|
|---|---|---|---|---|---|---|---|---|---|---|
||(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|(9)|(10)|
||Mid|End|Mid|End|Mid|End|Mid|End|Mid|End|
|In-Kind|0.239<sup>_∗∗∗_</sup>|0.156|0.129|0.121|0.235<sup>_∗∗∗_</sup>|0.184<sup>_∗_</sup>|0.202<sup>_∗∗_</sup>|0.183<sup>_∗_</sup>|0.274<sup>_∗∗∗_</sup>|0.130|
||(0.084)|(0.099)|(0.097)|(0.108)|(0.090)|(0.109)|(0.082)|(0.095)|(0.093)|(0.108)|
|Recognition|0.125|0.088|-0.037|0.053|0.224<sup>_∗∗_</sup>|0.130|0.102|0.118|0.146|0.060|
||(0.087)|(0.103)|(0.094)|(0.114)|(0.094)|(0.115)|(0.084)|(0.098)|(0.097)|(0.113)|
|Control mean|0.031|0.058|0.043|0.044|0.017|0.040|0.035|0.035|-0.004|-0.004|
|Observations|86840|57808|44105|28278|42735|29530|43421|28904|43419|28904|
|R-squared|0.097|0.121|0.124|0.121|0.084|0.109|0.109|0.153|0.097|0.102|
|Test:In-Kind=Recognition|0.205|0.514|0.092|0.577|0.908|0.647|0.256|0.511|0.199|0.543|

_Notes._ Each column represents a separate regression estimated using equation 1. The midline and endline results are in the oddand even-numbered columns respectively. The first two columns includes data for both grades and subjects, and includes grade and subject fixed effects. Columns (3)-(6) includes data for both subjects; columns (7)-(10) includes data for all grades. The final row reports the p-value of the F-test of equation of coefficients. Standard errors, clustered at a school level, are in parentheses. _∗p <_ 0 _._ 1, _∗∗ p <_ 0 _._ 05, _∗∗∗p <_ 0 _._ 01
Table 2: Impacts on learning— by teacher gender and identity

||Te|acher gen|der|Prefe|rred profe|ssion|
|---|---|---|---|---|---|---|
||(1)<br>Either|(2)<br>Female|(3)<br>Male|(4)<br>Either|(5)<br>Teacher|(6)<br>Other|
|In-Kind|0.230<sup>_∗∗_</sup>|0.194<sup>_∗_</sup>|0.189|0.255<sup>_∗∗∗_</sup>|0.284<sup>_∗∗_</sup>|0.220|
||(0.100)|(0.111)|(0.157)|(0.098)|(0.115)|(0.159)|
|Recognition|0.159|0.227<sup>_∗_</sup>|0.004|0.206<sup>_∗∗_</sup>|0.222<sup>_∗_</sup>|0.007|
||(0.105)|(0.125)|(0.156)|(0.104)|(0.116)|(0.189)|
|Observations|58949|47741|11208|46872|36048|10824|
|R-squared|0.099|0.115|0.183|0.141|0.157|0.225|
|Test:In-Kind=Recognition|0.501|0.780|0.250|0.633|0.592|0.252|

_Notes._ Each column represents a separate regression on the midline data, estimated using equation 1. Data in columns (1) to (3) are restricted to observations where all teachers in a given grade and school report the same gender. This allows us to unique identify the gender of the teacher. Columns (2) and (3) split this sample to female and male teachers, respectively. Similarly, columns (4) to (6) restrict the sample to schools and grades where all teachers either selected “teacher” or all teachers selected another profession to the question, asked at baseline: “if you could choose again, what profession would you choose?”. Columns (4) and (5) split this sample to teachers who selected “teacher” or not. Standard errors are in parentheses, and clustered at the school level. _∗p <_ 0 _._ 1, _∗∗ p <_ 0 _._ 05, _∗∗∗p <_ 0 _._ 01
Table 3: Impacts on learning— by student and teacher gender

|(1)<br>All teachers|(2)<br>Female|(3)<br>Male|
|---|---|---|
|In-Kind<br>0.272<sup>_∗∗∗_</sup>|0.210<sup>_∗_</sup>|0.265<sup>_∗_</sup>|
|(0.100)|(0.116)|(0.156)|
|Recognition<br>0.223<sup>_∗∗_</sup>|0.237<sup>_∗_</sup>|0.127|
|(0.100)|(0.121)|(0.149)|
|Boy<br>0.182<sup>_∗∗∗_</sup>|0.079<sup>_∗∗∗_</sup>|0.303<sup>_∗∗∗_</sup>|
|(0.041)|(0.029)|(0.064)|
|In-Kind x Boy<br>-0.072|-0.011|-0.146|
|(0.060)|(0.043)|(0.099)|
|Recognition x Boy<br>-0.124<sup>_∗∗_</sup>|-0.012|-0.211<sup>_∗∗_</sup>|
|(0.062)|(0.060)|(0.088)|
|Observations<br>58891|47707|11184|
|R-squared<br>0.098|0.111|0.184|
|_F-test (p-value): Impact on boys_|||
|In-Kind<br>0.065|0.074|0.478|
|Recognition<br>0.389|0.102|0.614|
|_F-test (p-value): In-Kind vs Recog_|_nition_||
|Girls<br>0.635|0.812|0.391|
|Boys<br>0.372|0.842|0.226|

_Notes._ Each column represents a separate regression on the midline data, estimated using equation 2. Data are restricted to same sample as columns (1) to (3) in Table 2. The bottom four rows report the p-values of the ˆ ˆ ˆ ˆ following respective F-tests: _α_ 1 + _α_ 3 = 0; _α_ 2 + _α_ 4 = 0 ˆ ˆ ˆ ˆ ˆ ˆ _α_ 1 = _α_ 2; _α_ 1 + _α_ 3 = _α_ 2 + _α_ 4. Standard errors are in parentheses. . _∗p <_ 0 _._ 1, _∗∗ p <_ 0 _._ 05, _∗∗∗p <_ 0 _._ 01 19
Table 4: Inspection-level outcomes at midline

||Cl|assroom q|uality||T|eaching qu|ality||
|---|---|---|---|---|---|---|---|---|
||(1)<br>Overall|(2)<br>Spatial<br>org.|(3)<br>Educ.<br>documents|(4)<br>Overall|(5)<br>Written<br>prep|(6)<br>Teaching<br>material|(7)<br>Class<br>practice|(8)<br>Refection|
|In-Kind|0.143<sup>_∗∗_</sup>|0.149<sup>_∗_</sup>|0.136<sup>_∗_</sup>|0.111<sup>_∗∗_</sup>|0.095|-0.022|0.115|0.256<sup>_∗∗∗_</sup>|
||(0.056)|(0.078)|(0.071)|(0.046)|(0.071)|(0.073)|(0.075)|(0.067)|
|Recognition|0.090|-0.020|0.200<sup>_∗∗∗_</sup>|0.079|0.210<sup>_∗∗∗_</sup>|-0.240<sup>_∗∗∗_</sup>|0.250<sup>_∗∗∗_</sup>|0.096|
||(0.060)|(0.087)|(0.073)|(0.051)|(0.075)|(0.078)|(0.079)|(0.078)|
|Control mean|0.000|0.000|0.000|0.000|0.000|0.000|0.000|0.000|
|Observations|1056|1056|1056|1056|1056|1056|1056|1056|
|R-squared|0.050|0.034|0.117|0.080|0.098|0.124|0.075|0.053|
|Test:In-Kind=Recognition|0.354|0.050|0.355|0.508|0.104|0.004|0.072|0.024|

_Notes._ Each column represents a separate regression using equation 1. Data is at a teacher level. The dependent variables in columns (2)-(3) and (5)-(8) are the scores assigned by inspectors for different domains of classroom and teaching quality, standardized to have a control mean of zero and standard deviation of one. The dependent variable in column (1) is the mean of the dependent variables in columns (2) and (3). The dependent variable in column (4) is a mean of the dependent variables in columns (5) to (8). Standard errors are in parentheses and clustered at the school level. *p _<_ 0 _._ 1, **p _<_ 0 _._ 05, ***p _<_ 0 _._ 01
Table 5: Impacts on learning— by exposure to Ebola

||Midline||Endline||
|---|---|---|---|---|
||(1)<br>Full|(2)<br>Full|(3)<br>Grade 3|(4)<br>Grade 4|
|In-Kind|0.242<sup>_∗∗∗_</sup>|0.227<sup>_∗∗_</sup>|0.129|0.294<sup>_∗∗_</sup>|
||(0.088)|(0.101)|(0.124)|(0.124)|
|Recognition|0.109|0.171|0.129|0.213<sup>_∗_</sup>|
||(0.093)|(0.106)|(0.136)|(0.128)|
|Ebola|-0.001|-0.000|0.000|-0.000|
||(0.002)|(0.002)|(0.002)|(0.002)|
|Ebola x In-Kind|0.000|-0.006<sup>_∗∗_</sup>|-0.004|-0.007<sup>_∗∗_</sup>|
||(0.003)|(0.002)|(0.003)|(0.003)|
|Ebola x Recognition|0.001|-0.006<sup>_∗∗∗_</sup>|-0.005<sup>_∗∗_</sup>|-0.008<sup>_∗∗∗_</sup>|
||(0.002)|(0.002)|(0.002)|(0.003)|
|Observations|86578|57444|28146|29298|
|R-squared|0.124|0.143|0.149|0.154|

_Notes._ Each column is a separate regression including the same controls as in equation 1. The variable “Ebola” is the total number of deaths reported in a prefecture by the time of endline data collection, May 2014. The WHO only reported cases for 29 out 33 prefectures. We assume that there were no cases for the four prefectures with no data, but results are the same if instead we treat these prefectures as having missing data (Table A.15).

21
# **A ONLINE APPENDIX**

## **A.1 Determination of Rewards and the Reward Ceremonies**

Table A.1 shows the mapping between the performance indicator and rewards. The rewards ranged from bronze to platinum. There was one national ceremony (for recipients of the platinum certificate), one ceremony in each of the eight regions (for the gold certificate), one ceremony in each of the 38 prefectures (silver), and one ceremony in each of the 243 sub-prefectures (bronze). The ministry of education organized the national ceremony; the Regional Authorities of Education organized the regional and prefecture-level ceremonies; and the “School Delegation of Elementary Education” (DSEE) organized the sub-prefecture ceremonies. The ceremonies were attended by government officials, trade unions, national or local press, locally elected officials, and school parent associations.

Table A.2 shows the breakdown of awards received after the first year of the program and the value of the awards as a proportion of the average civil servant teacher salary. Over a third did not receive any reward, and a quarter of teachers received the highest possible reward. The value of the reward was highly convex, with 4% of salary for the lowest tier (rice and radio), and almost 49% of a teacher’s annual salary for the highest tier (rice, TV, cell phone, and generator).

## **A.2 Tests for Internal Validity**

### **A.2.1 Balance and Attrition**

We investigate internal validity by comparing baseline data from Grade 2 and Grade 3 students from the student assessment, as well as school-level data as reported by school principals, across the groups. The sample is balanced on key variables–test scores and gender–collected at the baseline; only age shows a difference between control and treatment arms (Table A.7). School characteristics (number of Grade 2 and Grade 3 students, headteacher gender and age) are likewise never statistically significantly different across the groups, with the exception of headteacher age which is very slightly higher in the control group than in the recognition group. These differences are consistent with random chance. The sample is also balanced by exposure to Ebola. We further test for baseline balance for the sample of schools for which we have midline and endline data, respectively. As before, there are very few systematic differences across the groups in these schools (Tables A.9 and A.10). Two comparisons are statistically significant at the 10 percent level–but overall this small number is still consistent with pure chance.

While balance at baseline establishes the internal validity of the experimental design, 1
balance at midline and endline also suggest that school attrition is not selective. We further investigates whether there are systematic differences across groups in whether students in specific grades were administered the learning assessment during the midline and endline surveys. We do this by regressing a zero/one variable indicating whether specific data appear in the survey on indicators for the treatment groups, with the control group being the left out/reference category. The results show that there are no systematic differences across groups, again suggesting the results are not likely to be biased by which data are available for the analysis sample (Table A.8).

In sum, the samples we are using in our analysis are indeed balanced across the randomly assigned groups at baseline. In addition, attrition of schools and student assessments from the sample is not selective with respect to the treatment groups. Together these findings, consistent with the intent of the experimental design, allow us to compare indicators and outcomes at midline and endline, and confidently attribute any differences to the intervention itself.

# **A.2.2 Investigating Gaming by Teachers**

High-stakes accountability schemes can produce perverse effects such as reducing the number of students tested in order to increase performance measures. We do not have data that track individual students over time; however, we examine enrollment numbers to explore whether fewer students are being assessed or enrolled, or if more students drop out, in the treatment groups relative to the control. Table A.11 regresses log number of students assessed for the respective grades at midline and endline on treatment dummies, including strata fixed effects. There is no evidence of a strategic holding back of students for either treatment arm, grade, or year of data collection. In fact, slightly more students were assessed for both grades and rounds of data collection for the In-Kind arm ( _p_ = 0 _._ 116 and 0 _._ 726 for grades 3 and 4 at midline; and _p_ = 0 _._ 267 and 0 _._ 111 at endline). The results for the Recognition arm are also always statistically insignificant ( _p_ = 0 _._ 349 and 0 _._ 762 for grades 3 and 4 respectively at midline; and _p_ = 0 _._ 757 and 0 _._ 87 at endline). It is not clear why the number of students assessed is relative larger in the In-Kind arm; it could be that the learning gains also reduces dropouts. The number of grade 2 students assessed at midline is also larger by a similar magnitude in the In-Kind treatment, even though grade 2 student scores did not determine pay-outs, suggesting that this pattern is not due to gaming. The increase in the number of students assessed is partly driven by outliers. Table A.13 shows that our main results on student learning are robust to dropping these students from the sample.

Other forms of gaming include cheating and “teaching to the test”. We believe that former is unlikely, given the efforts taken to minimize opportunities for cheating or collusion 2
(see sub-section 2.1). The latter also seems unlikely, since there was no prior knowledge about the universe of potential test items, and different students received different test booklets. Moreover, “teaching to the test” is not necessarily a bad thing provided that the potential test items are a comprehensive reflection of the learning goals required for a given subject/grade.

# **A.3 Additional Tables and Figures**

3
Table A.1: Rewards and performance indicator level

|Performance<br>indicator value|In-Kind<br>Reward|Recognition<br>Reward|
|---|---|---|
|0.4 _<_ x _≤_1|One bag of rice<br>and a radio|Bronze certifcate and ceremony<br>at the community level|
|1 _<_ x _≤_1.6|One bag of rice, a radio,<br>a cell phone, a TV|Silver certifcate and ceremony<br>at the community and prefectural level|
|1.6 _<_ x _≤_2.4|One bag of rice, a radio,<br>a cell phone, a TV|Gold certifcate and ceremony<br>at the community, prefectural,<br>and regional level|
|x _>_ 2.4|One bag of rice, a radio,<br>a cell phone, a TV,<br>a generator|Platinum certifcate and ceremony<br>at the community, prefectural, regional and<br>national level|

4
Table A.2: Distribution of In-Kind rewards

||Proportion of<br>teachers|Proportion of<br>annual salary|
|---|---|---|
|None|36%|0%|
|Rice & radio|18%|4%|
|Rice & radio & cellphone|10%|8%|
|Rice & TV & cellphone|11%|32%|
|Rice & TV & cellphone & generator|25%|49%|

Table A.3: Quality of Implementation (teacher data)

||(1)|(2)<br>**Midlin**|(3)<br>**e**|(4)|(5)|(6)<br>**Endli**|(7)<br>**ne**|(8)|
|---|---|---|---|---|---|---|---|---|
||Control|In-Kind|Recog.|Obs|Control|In-Kind|Recog.|Obs|
|School participates|0.967|0.972|0.964|1055|0.788|0.918|0.832|1100|
|_Which teachers targeted?_|||||||||
|Only grade 2|0.032|0.051|0.065|999|0.104|0.111|0.115|861|
|Only grade 3|0.388|0.369|0.376|1016|0.296|0.157|0.254|861|
|Only grade 4|0.386|0.367|0.387|1023|0.052|0.071|0.062|861|
|Grades 3 and 4|0.704|0.712|0.753|1023|0.341|0.519|0.384|864|
|_Type of award_|||||||||
|Financial|0.484|0.355|0.462|1000|0.129|0.098|0.174|864|
|In-Kind|0.721|0.916|0.794|1023|0.414|0.739|0.217|864|
|Certifcate|0.455|0.497|0.620|1005|0.169|0.115|0.504|864|
|Ceremony|0.204|0.179|0.331|1002|0.064|0.056|0.244|864|
|_Performance metric_|||||||||
|Student test scores|0.756|0.840|0.794|1019|0.863|0.932|0.790|863|
|Inspection score|0.943|0.956|0.929|1022|0.860|0.921|0.795|865|
|French evaluated|0.943|0.947|0.969|1017|0.976|0.986|0.964|897|
|Math evaluated|0.931|0.924|0.959|1017|0.960|0.989|0.957|895|
|Science evaluated|0.334|0.337|0.376|993|0.195|0.123|0.148|779|
|_Abs vs relative performance_|||||||||
|Relative—Other teachers in school|0.238|0.181|0.184|1027|0.383|0.295|0.304|906|
|Relative—Other schools|0.158|0.164|0.146|1027|0.199|0.187|0.204|899|
|_Receipt of award_|||||||||
|Teacher received|||||0.066|0.458|0.504|889|
|Money|||||0.012|0.067|0.126|845|
|In-Kind|||||0.008|0.387|0.077|868|
|Certifcate|||||0.008|0.027|0.386|851|
|Ceremony|||||0.008|0.030|0.125|847|
|School received|||||0.058|0.817|0.754|936|

5
Table A.4: Data collection instruments

|Instruments|Sections|
|---|---|
|Principal questionnaire|A. School and principal identifcation|
||B. Demographics|
||C. Education and professional training|
||D. Work experience and training needs<br>E. Pedagogical practices and languages|
||F. School basic characteristics<br>G. School environment|
||H. Interaction with colleagues (subordinate, supervisors, etc.)<br>I. Support and monitoring of teachers<br>J. Motivation|
|Teacher questionnaire|A. Class and teacher identifcation<br>B. Demographics|
||C. Class characteristics|
||D. Education and professional training|
||E. Work experience and training needs|
||F. Pedagogical practices and languages|
||G. Interaction with colleagues<br>H. Motivation|
||I. Absenteeism and event disturbing teaching|
||J. Remuneration<br>K. Perception of key factors infuencing student learning|
||L. Performance recognition or punishment|
|Student tests|A. Identifcation of school, class, and teachers|
||B. School-related student characteristics|
||C. Student environmental and familial backgrounds|
||D. French test questions|
||E. Math test questions|
|Inspection bulletin|A. Class and inspector identifcation|
||B. Teacher identifcation<br>C. Summary of scores|
||D. General material and spatial classroom arrangement|
||1. Lesson 1 – Identifcation of the lesson|
||2. Lesson 1 – Teaching and learning material preparation|
||3. Lesson 1 – Lesson planning (Competency-based approach)|
||4. Lesson 1 – Delivery of the lesson|
||5. Lesson 1 – Analysis of own performance|
||1. Lesson 2 – Identifcation of the lesson|
||2. Lesson 2 – Teaching and learning material preparation|
||3. Lesson 2 – Lesson planning (Competency-based approach)|
||4. Lesson 2 – Delivery of the lesson|
||5. Lesson 2 – Analysis of own performance|

6
Table A.5: Sample sizes for different rounds and instruments

|Variable|2012<br>Baseline|2013<br>Midline|2014<br>Endline|
|---|---|---|---|
|No of Schools (principal data)|416|403|390|
|No of Schools (teacher data)|406|387|360|
|No of Schools (student data)|408|417|391|
|No of Schools (inspection data)||387|388|
|No of Teachers (teacher data)|1,165|1,068|1,120|
|No of Teachers (inspection data)||1,057|1,179|
|No of Grade 2 Students|19,621|12,327|—|
|No of Grade 3 Students|18,663|22,053|21,368|
|No of Grade 4 Students|—|14,139|14,765|

Table A.6: Teacher characteristics

|Variable|N|(1)<br>Male<br>Mean/SE|N|(2)<br>Female<br>Mean/SE|N|(3)<br>Total<br>Mean/SE|T-test<br>Diference<br>(1)-(2)|
|---|---|---|---|---|---|---|---|
|Class size|377|43.363<br>(0.900)|755|52.375<br>(0.558)|1132|49.374<br>(0.494)|-9.011***|
|Permanent|386|0.902<br>(0.015)|784|0.955<br>(0.007)|1170|0.938<br>(0.007)|-0.054***|
|Insufcient— recognition|385|0.249<br>(0.022)|784|0.281<br>(0.016)|1169|0.270<br>(0.013)|-0.031|
|Insufcient— salary|385|0.675<br>(0.024)|784|0.759<br>(0.015)|1169|0.731<br>(0.013)|-0.084***|
|Monthly salary (USD)|340|125.268<br>(1.254)|743|115.562<br>(0.598)|1083|118.610<br>(0.585)|9.706***|
|Choose teaching|381|0.635<br>(0.025)|775|0.732<br>(0.016)|1156|0.700<br>(0.013)|-0.096***|

_Notes_ : The value displayed for t-tests are the differences in the means between male and female teachers, using baseline data. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

7
Table A.7: Baseline balance on key variables

||(|1)|(|2)||(3)|T-|test|
|---|---|---|---|---|---|---|---|---|
||Co|ntrol|In-|Kind|Reco|gnition|Dife|rence|
|Variable|N|Mean|N|Mean|N|Mean|(1)-(2)|(1)-(3)|
|_Student-level (grade 2)_|||||||||
|Math|6466<br>[112]|0.027<br>(0.075)|6643<br>[109]|0.078<br>(0.067)|6512<br>[115]|-0.089<br>(0.078)|-0.051|0.117|
|Language|6466<br>[112]|0.032<br>(0.069)|6643<br>[109]|0.057<br>(0.067)|6512<br>[115]|-0.013<br>(0.064)|-0.025|0.045|
|Female|6454<br>[112]|0.474<br>(0.008)|6639<br>[109]|0.477<br>(0.009)|6495<br>[115]|0.471<br>(0.008)|-0.003|0.003|
|Age|6262<br>[112]|9.411<br>(0.067)|6398<br>[109]|9.561<br>(0.092)|6299<br>[115]|9.585<br>(0.079)|-0.150**|-0.175**|
|_Student-level (grade 3)_|||||||||
|Math|6089<br>[119]|-0.004<br>(0.077)|6280<br>[120]|-0.055<br>(0.067)|6294<br>[117]|-0.115<br>(0.077)|0.051|0.112|
|Language|6089<br>[119]|0.035<br>(0.077)|6280<br>[120]|-0.007<br>(0.063)|6294<br>[117]|-0.091<br>(0.084)|0.041|0.126|
|Female|6083<br>[119]|0.473<br>(0.009)|6267<br>[120]|0.472<br>(0.009)|6290<br>[117]|0.480<br>(0.008)|0.001|-0.007|
|Age|5964<br>[118]|10.949<br>(0.071)|6143<br>[120]|11.071<br>(0.078)|6173<br>[117]|11.037<br>(0.067)|-0.122|-0.088|
|_School-level_|||||||||
|No. grade 2 students|112|57.732<br>(3.545)|109|60.945<br>(4.444)|115|56.626<br>(3.600)|-3.213|1.106|
|No. grade 3 students|119|51.168<br>(3.130)|120|52.333<br>(4.200)|117|53.795<br>(3.812)|-1.165|-2.627|
|Headteacher female|138|0.181<br>(0.033)|139|0.165<br>(0.032)|139|0.230<br>(0.036)|0.016|-0.049|
|Headteacher age|132|48.455<br>(0.782)|136|47.882<br>(0.846)|135|46.281<br>(0.913)|0.572|2.173**|

_Notes_ : The value displayed for t-tests are the differences in the means across the groups. Standard errors, in parenthesis are clustered at the school level. Observations with square brackets indicate number of clusters. Strata fixed effects included in all estimation regressions. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

8
Table A.8: Attrition

|||Midline||End|line|
|---|---|---|---|---|---|
||(1)<br>Grade 2|(2)<br>Grade 3|(3)<br>Grade 4|(4)<br>Grade 3|(5)<br>Grade 4|
|In-Kind|-0.006|-0.042|0.022|-0.031|0.069|
||(0.053)|(0.045)|(0.044)|(0.049)|(0.044)|
|Recognition|0.011|-0.051|-0.022|-0.022|-0.011|
||(0.054)|(0.047)|(0.044)|(0.048)|(0.048)|
|Control mean|0.691|0.831|0.838|0.772|0.794|
|Observations|408|408|408|408|408|
|R-squared|0.129|0.109|0.116|0.170|0.097|
|Test:In-Kind=Recognition|0.759|0.856|0.291|0.859|0.078|

_Notes._ Each column represents a separate regression, including strata fixed effects. Data is at a schools level, and the dependent variable is an indicator variable equal to one if student learning data is available for a given school in a given grade and a given round of data collection. Standard errors are in parentheses. . _∗p <_ 0 _._ 1, _∗∗ p <_ 0 _._ 05, _∗∗∗p <_ 0 _._ 01 9
Table A.9: Baseline balance, restricted to sample with midline student assessment data

|||(1)||(2)||(3)|T-|test|
|---|---|---|---|---|---|---|---|---|
||Co|ntrol|In-|Kind|Reco|gnition|Dif|erence|
|Variable|N|Mean|N|Mean|N|Mean|(1)-(2)|(1)-(3)|
|_Student-level (grade 2)_|||||||||
|Math|5528<br>[84]|0.049<br>(0.084)|5771<br>[80]|0.093<br>(0.075)|5409<br>[86]|-0.100<br>(0.088)|-0.044|0.149|
|Language|5528<br>[84]|0.050<br>(0.076)|5771<br>[80]|0.063<br>(0.074)|5409<br>[86]|-0.025<br>(0.074)|-0.013|0.075|
|Female|5517<br>[84]|0.482<br>(0.008)|5767<br>[80]|0.484<br>(0.010)|5400<br>[86]|0.479<br>(0.009)|-0.003|0.003|
|Age<br>_Student-level (grade 3)_|5349<br>[84]|9.485<br>(0.071)|5549<br>[80]|9.618<br>(0.101)|5255<br>[86]|9.644<br>(0.085)|-0.133*|-0.159**|
|Math|5473<br>[96]|0.048<br>(0.082)|5642<br>[93]|-0.032<br>(0.071)|5482<br>[91]|-0.122<br>(0.085)|0.080|0.171|
|Language|5473<br>[96]|0.067<br>(0.083)|5642<br>[93]|0.017<br>(0.067)|5482<br>[91]|-0.098<br>(0.094)|0.050|0.165|
|Female|5468<br>[96]|0.479<br>(0.009)|5633<br>[93]|0.477<br>(0.010)|5478<br>[91]|0.478<br>(0.009)|0.003|0.001|
|Age|5364<br>[95]|10.977<br>(0.075)|5545<br>[93]|11.112<br>(0.082)|5373<br>[91]|11.096<br>(0.069)|-0.136*|-0.119|
|_School-level_|||||||||
|No. grade 2 students|112|57.732<br>(3.545)|109|60.945<br>(4.444)|115|56.626<br>(3.600)|-3.213|1.106|
|No. grade 3 students|119|51.168<br>(3.130)|118|52.907<br>(4.251)|116|54.112<br>(3.832)|-1.739|-2.944|
|Headteacher female|138|0.181<br>(0.033)|137|0.168<br>(0.032)|138|0.232<br>(0.036)|0.013|-0.051|
|Headteacher age|132|48.455<br>(0.782)|134|47.754<br>(0.854)|134|46.388<br>(0.913)|0.701|2.066**|
|No. deaths, Ebola|107|14.523<br>(4.002)|105|11.638<br>(3.497)|108|10.787<br>(3.111)|2.885|3.736|

_Notes_ : The value displayed for t-tests are the differences in the means across the groups. Standard errors, in parenthesis are clustered at the school level. Observations with square brackets indicate number of clusters. Strata fixed effects included in all estimation regressions. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

10
Table A.10: Baseline balance, restricted to sample with endline student assessment data

||(|1)|(|2)||(3)|T|-test|
|---|---|---|---|---|---|---|---|---|
||Co|ntrol|In-|Kind|Reco|gnition|Dif|erence|
|Variable|N|Mean|N|Mean|N|Mean|(1)-(2)|(1)-(3)|
|_Student-level (grade 2)_|||||||||
|Math|6346<br>[109]|0.019<br>(0.076)|6643<br>[109]|0.078<br>(0.067)|6311<br>[112]|-0.109<br>(0.078)|-0.059|0.129|
|Language|6346<br>[109]|0.024<br>(0.070)|6643<br>[109]|0.057<br>(0.067)|6311<br>[112]|-0.030<br>(0.064)|-0.033|0.055|
|Female|6334<br>[109]|0.476<br>(0.008)|6639<br>[109]|0.477<br>(0.009)|6294<br>[112]|0.469<br>(0.008)|-0.001|0.007|
|Age|6153<br>[109]|9.423<br>(0.067)|6398<br>[109]|9.561<br>(0.092)|6101<br>[112]|9.563<br>(0.075)|-0.138*|-0.140**|
|_Student-level (grade 3)_|||||||||
|Math|5846<br>[111]|0.018<br>(0.079)|6129<br>[113]|-0.050<br>(0.068)|5932<br>[107]|-0.117<br>(0.077)|0.069|0.136|
|Language|5846<br>[111]|0.051<br>(0.079)|6129<br>[113]|-0.003<br>(0.063)|5932<br>[107]|-0.106<br>(0.085)|0.054|0.157|
|Female|5840<br>[111]|0.479<br>(0.009)|6116<br>[113]|0.473<br>(0.009)|5928<br>[107]|0.479<br>(0.008)|0.006|0.000|
|Age|5727<br>[110]|10.968<br>(0.071)|5996<br>[113]|11.082<br>(0.079)|5815<br>[107]|11.044<br>(0.067)|-0.115|-0.076|
|_School-level_|||||||||
|No. grade 2 students|109|58.220<br>(3.631)|109|60.945<br>(4.444)|112|56.348<br>(3.563)|-2.725|1.872|
|No. grade 3 students|111|52.667<br>(3.303)|113|54.239<br>(4.396)|107|55.439<br>(3.945)|-1.572|-2.773|
|Headteacher female|129|0.194<br>(0.035)|132|0.174<br>(0.033)|127|0.244<br>(0.038)|0.020|-0.050|
|Headteacher age|123|49.008<br>(0.787)|129|47.961<br>(0.877)|123|46.976<br>(0.922)|1.047|2.033*|
|No. deaths, Ebola|98|15.224<br>(4.347)|101|11.446<br>(3.624)|99|11.606<br>(3.381)|3.779|3.618|

_Notes_ : The value displayed for t-tests are the differences in the means across the groups. Standard errors, in parenthesis are clustered at the school level. Observations with square brackets indicate number of clusters. Strata fixed effects included in all estimation regressions. ***, **, and * indicate significance at the 1, 5, and 10 percent critical level.

11
Table A.11: Log number of students assessed— by grade and round of data collection

||Bas|eline||Midline||End|line|
|---|---|---|---|---|---|---|---|
||(1)<br>Grade 2|(2)<br>Grade 3|(3)<br>Grade 2|(4)<br>Grade 3|(5)<br>Grade 4|(6)<br>Grade 3|(7)<br>Grade 4|
|In-Kind|0.030|-0.051|0.148|0.145|0.113|0.026|0.106|
||(0.076)|(0.072)|(0.134)|(0.092)|(0.101)|(0.073)|(0.066)|
|Recognition|0.004|0.002|0.172|0.093|-0.034|-0.020|0.012|
||(0.070)|(0.068)|(0.126)|(0.099)|(0.109)|(0.064)|(0.071)|
|Control mean|57.732|51.168|39.835|57.888|52.508|43.269|41.387|
|Observations|336|356|289|336|350|317|340|
|R-squared|0.340|0.399|0.330|0.406|0.388|0.412|0.433|
|Test:In-Kind=Recognition|0.720|0.458|0.846|0.601|0.196|0.539|0.168|

_Notes._ Each column represents a separate regression, including strata fixed effects. The dependent variable is the log of the the total number of students assessed for a given school and grade, and for a given round of data collection. Data is at a school level. Standard errors are in parentheses. . _∗p <_ 0 _._ 1, _∗∗ p <_ 0 _._ 05, _∗∗∗p <_ 0 _._ 01 12
Table A.12: Breakdown of per student costs by treatment arm

||**In-kind**|**Recognition**|
|---|---|---|
|Data Collection|$ 5.57|$ 5.57|
|Award Ceremonies|N/A|$ 2.72|
|In-kind Rewards|$ 6.18|N/A|
|**Total**|$ 11.75|$ 8.29|

_Notes._ Costs are in USD, 2014 prices. Data collection costs are from 2014 (endline), when a random sample of students in every school were assessed . Implementation costs data are from 2013 (midline) but adjusting for one year of inflation (the award ceremonies did not take place in 2014). We do not include the costs of the communication campaign conducted prior to the start of the program, since this is a fixed cost and future communication can take place in combination with the award ceremonies. The schools in our sample have on average 77 students enrolled in both grade 3 and grade 4.

13
Table A.13: Main impacts on learning, dropping outliers

||Full s|ample|Gra|de 3|Grad|e 4|Fre|nch|Ma|th|
|---|---|---|---|---|---|---|---|---|---|---|
||(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|(9)|(10)|
||Mid|End|Mid|End|Mid|End|Mid|End|Mid|End|
|In-Kind|0.239<sup>_∗∗∗_</sup>|0.156|0.129|0.121|0.235<sup>_∗∗∗_</sup>|0.184<sup>_∗_</sup>|0.202<sup>_∗∗_</sup>|0.183<sup>_∗_</sup>|0.274<sup>_∗∗∗_</sup>|0.130|
||(0.084)|(0.099)|(0.097)|(0.108)|(0.090)|(0.109)|(0.082)|(0.095)|(0.093)|(0.108)|
|Recognition|0.125|0.088|-0.037|0.053|0.224<sup>_∗∗_</sup>|0.130|0.102|0.118|0.146|0.060|
||(0.087)|(0.103)|(0.094)|(0.114)|(0.094)|(0.115)|(0.084)|(0.098)|(0.097)|(0.113)|
|Control mean|0.031|0.058|0.043|0.044|0.017|0.040|0.035|0.035|-0.004|-0.004|
|Observations|86840|57808|44105|28278|42735|29530|43421|28904|43419|28904|
|R-squared|0.097|0.121|0.124|0.121|0.084|0.109|0.109|0.153|0.097|0.102|
|Test:In-Kind=Recognition|0.205|0.514|0.092|0.577|0.908|0.647|0.256|0.511|0.199|0.543|

_Notes._ See Table 1. Students in grades with more than 300 students are dropped from the sample.
<!-- Start of picture text --> 2012 2013 2014<br>ve May Oct Nov Dec va May cal oe Dec Jan May<br>Data collection<br>Pilot<br>Student assessment<br>Inspection<br>Teacher questionnaire<br>Implementation<br>Communication campaign<br>Reward distribution<br>School year i<br>Ebola | ot tT tT<br><!-- End of picture text -->
<!-- Start of picture text --> 6-<br>Aq<br>2<<br>o<br>Wws<br>27<br>o- U 0 I I I 0 j<br>0 50 100 150 200<br>Known cases of Ebola<br><!-- End of picture text -->