metadata: authors: 0: author: Leaver,Clare
1: author: Ozier,Owen
2: author: Serneels,Pieter Maria
3: author: Zeitlin,Andrew
count: Rwanda
docna: 0: docna: Recruitment, Effort, and Retention Effects
            of Performance Contracts for Civil Servants : Experimental
            Evidence from Rwandan Primary Schools
docty: Policy Research Working Paper
subtopic: Educational Institutions & Facilities,Effective Schools and Teachers,Rural Labor Markets,Health Economics & Finance,Labor Markets,Disability
teratopic: Social Protections and Labor,Rural Development,Health, Nutrition and Population,Education
abstracts: cdata!: This paper reports on a two-tiered
            experiment designed to separately identify the selection and
            effort margins of pay-for-performance (P4P). At the
            recruitment stage, teacher labor markets were randomly
            assigned to a 'pay-for-percentile' or fixed-wage
            contract. Once recruits were placed, an unexpected,
            incentive-compatible, school-level re-randomization was
            performed, so that some teachers who applied for a
            fixed-wage contract ended up being paid by P4P, and vice
            versa. By the second year of the study, the within-year
            effort effect of P4P was 0.16 standard deviations of pupil
            learning, with the total effect rising to 0.20 standard
            deviations after allowing for selection.
geo_reg_key: 517196,517191
repnme: repnme: Recruitment, Effort, and Retention Effects
            of Performance Contracts for Civil Servants : Experimental
            Evidence from Rwandan Primary Schools
display_title: Recruitment, Effort, and Retention
            Effects of Performance Contracts for Civil Servants :
            Experimental Evidence from Rwandan Primary Schools
listing_relative_url: /research/2020/09/32397313/recruitment-effort-retention-effects-performance-contracts-civil-servants-experimental-evidence-rwandan-primary-schools
topicv3: Teachers Management m1326119 1523,Results-Based Financing for Health
            m1326324 1298,Labor Markets m1326488 855,Jobs m1503931 2670,Blindness m1582294 4218
publishtoextweb_dt: 2020-09-11T00:00:00Z
historic_topic: Social Protections and Labor,Rural Development,Health, Nutrition and Population,Education
origu: DECRG: Human Development (DECHD)
no_of_pages: 82
author_keys: 0: author_key: 691398
1: author_key: 320793
2: author_key: 925890
model_extractions: Policy Research Working Paper 9395 Recruitment, Effort, and Retention Effects of Performance Contracts for Civil Servants Experimental Evidence from Rwandan Primary Schools _Clare Leaver Owen Ozier Pieter Serneels Andrew Zeitlin_ Development Economics Development Research Group September 2020 A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click **here** for direct access.
Policy Research Working Paper 9395

# **Abstract**

This paper reports on a two-tiered experiment designed to separately identify the selection and effort margins of pay-for-performance (P4P). At the recruitment stage, teacher labor markets were randomly assigned to a ‘pay-for-percentile’ or fixed-wage contract. Once recruits were placed, an unexpected, incentive-compatible, school-level re-randomization was performed, so that some teachers who applied for a fixed-wage contract ended up being paid by P4P, and vice versa. By the second year of the study, the within-year effort effect of P4P was 0.16 standard deviations of pupil learning, with the total effect rising to 0.20 standard deviations after allowing for selection.

This paper is a product of the Development Research Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at clare.leaver@bsg.ox.ac.uk, owen.ozier@williams.edu, p.serneels@uea.ac.uk, and az332@georgetown.edu. A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click **here** for direct access.

<!-- Start of picture text --> TRANSPARENT<br>ANALYSIS<br>G<br>R E<br>N<br>S<br>I P<br>Y E<br>K A<br>C A<br>R P<br>LI R<br>O E<br>O C<br>W R<br>H<br>S<br>P<br><!-- End of picture text --> _The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
Recruitment, effort, and retention effects of performance contracts for civil servants: Experimental evidence from Rwandan primary schools Clare Leaver,<sup>_∗_</sup> Owen Ozier,<sup>_†_</sup> Pieter Serneels,<sup>_‡_</sup> and Andrew Zeitlin<sup>_§¶_</sup> JEL codes: C93, I21, J45, M52, O15.

Keywords: pay-for-performance, selection, incentives, teachers, field experiment.

> _∗_ Blavatnik School of Government, University of Oxford; and CEPR (clare.leaver@bsg.ox.ac.uk)

> _†_ Deparment of Economics, Williams College; World Bank Development Research Group; BREAD; and IZA (owen.ozier@williams.edu)

> _‡_ School of International Development, University of East Anglia; EGAP; and IZA (p.serneels@uea.ac.uk)

> _§_ McCourt School of Public Policy, Georgetown University; and CGD (az332@georgetown.edu)

> _¶_ We thank counterparts at REB and MINEDUC for advice and collaboration, and David Johnson for help with the design of student and teacher assessments. We are grateful to Katherine Casey, Jasper Cooper, Ernesto Dal B ́o, Erika Deserranno, David Evans, Dean Eckles, Frederico Finan, James Habyarimana, Caroline Hoxby, Macartan Humphreys, Pamela Jakiela, Julien Labonne, David McKenzie, Ben Olken, Berk Ozler,<sup> ̈</sup> Cyrus Samii, Kunal Sen, Martin Williams, and audiences at BREAD, DfID, EDI, NBER, SIOE, and SREE for helpful comments. IPA staff members Kris Cox, Stephanie De Mel, Olive Karekezi Kemirembe, Doug Kirke-Smith, Emmanuel Musafiri, and Phillip Okull, and research assistants Claire Cullen, Robbie Dean, Ali Hamza, Gerald Ipapa, and Saahil Karpe all provided excellent support. Financial support was provided by the U.K. Department for International Development (DfID) via the International Growth Centre and the Economic Development and Institutions Programme, by Oxford University’s John Fell Fund, and by the World Bank’s SIEF and REACH trust funds. Leaver is grateful for the hospitality of the Toulouse School of Economics, 2018–2019. We received IRB approval from the Rwanda National Ethics Committee and from Innovations for Poverty Action. This study is registered as AEA RCT Registry ID AEARCTR-0002565. The findings in this paper are the opinions of the authors, and do not represent the opinions of the World Bank, its Executive Directors, or the governments they represent. All errors and omissions are our own.
# **1 Introduction**

The ability to recruit, elicit effort from, and retain civil servants is a central issue for any government. This is particularly true in a sector such as education where people—that is, human rather than physical resources—play a key role. Effective teachers generate private returns for students through learning gains, educational attainment, and higher earnings (Chetty et al., 2014a,b), as well as social returns through improved labor-market skills that drive economic growth (Hanushek and Woessmann, 2012). And yet in varying contexts around the world, governments struggle to maintain a skilled and motivated teacher workforce (Bold et al., 2017).

One policy option in this context is _pay-for-performance_ . These compensation schemes typically reward teacher inputs such as presence and conduct in the classroom, teacher value added based on student learning, or both (see, e.g., Muralidharan and Sundararaman, 2011b). In principle, they can address the difficulty of screening for teacher quality ex ante (Staiger and Rockoff, 2010), as well as the limited oversight of teachers on the job (Chaudhury et al., 2006).

Yet pay-for-performance divides opinion. Critics, drawing upon public administration, social psychology, and behavioral economics, argue that pay-for-performance could dampen the effort of workers (B ́enabou and Tirole, 2003; Deci and Ryan, 1985; Krepps, 1997). Concerns are that such schemes may: recruit the wrong types, individuals who are “in it for the money”; lower effort by eroding intrinsic motivation; and fail to retain the right types because good teachers become de-motivated and quit. By contrast, proponents point to classic contract theory (Lazear, 2003; Rothstein, 2015) and evidence from private-sector jobs with readily measurable output (Lazear, 2000) to argue that pay-for-performance will have positive effects on both compositional and effort margins. Under this view, such schemes: recruit the right types, individuals who anticipate performing well in the classroom; raise effort by strengthening extrinsic motivation; and retain the right types because good teachers feel rewarded and stay put.

This paper conducts the first prospective, randomized controlled trial designed to identify both the compositional and effort margins of pay-for-performance. A novel, two-tiered experiment separately identifies these effects. This is combined with detailed data on applicants to jobs, the skills and motivations of actual hires, and their performance over two years on the job, to evaluate the effects of pay-forperformance on the recruitment, effort, and retention of civil servant teachers.

1
At the center of this study is a pay-for-performance (hereafter P4P) contract, designed jointly with the Rwanda Education Board and Ministry of Education. Building on extensive consultations and a pilot year, this P4P contract rewards the top 20 percent of teachers with extra pay using a metric that equally weights learning outcomes in teachers’ classrooms alongside three measures of teachers’ inputs into the classroom (presence, lesson planning, and observed pedagogy). The measure of learning used was based on a _pay-for-percentile_ scheme that makes student performance at all levels relevant to teacher rewards (Barlevy and Neal, 2012). The tournament nature of this contract allows us to compare it to a fixed-wage (hereafter FW) contract that is equal in expected payout.

Our two-tiered experiment first randomly assigns labor markets to either P4P or FW _advertisements_ , and then uses a surprise re-randomization of _experienced_ contracts at the school level to enable estimation of pure compositional effects within each realized contract type. The first stage was undertaken during recruitment for teacher placements for the 2016 school year. Teacher labor markets are defined at the district by subject-family level. We conducted the experiment in six districts (18 labor markets) which, together, cover more than half the upper-primary teacher hiring lines for the 2016 school year. We recruited into the study all primary schools that received such a teacher to fill an upper-primary teaching post (a total of 164 schools). The second stage was undertaken once 2016 teacher placements had been finalized. Here, we randomly re-assigned each of these 164 study schools in their entirety to either P4P or FW contracts; all teachers who taught core-curricular classes to upper-primary students, including both newly placed recruits and incumbents, were eligible for the relevant contracts. We offered a signing bonus to ensure that no recruit, regardless of her belief about the probability of winning, could be made worse off by the re-randomization and, consistent with this, no one turned down their (re-)randomized contract. As advertised at the time of recruitment, incentives were in place for two years, enabling us to study retention as well as to estimate higher-powered tests of effects using outcomes from both years.

Our three main findings are as follows. First, on recruitment, advertised P4P contracts did not change the distribution of measured teacher skill either among applicants in general or among new hires in particular. This is estimated sufficiently precisely to rule out even small negative effects of P4P on measured skills. Advertised P4P contracts did, however, select teachers who contributed less in a framed Dictator Game played at baseline to measure intrinsic motivation. In spite of this, 2
teachers recruited under P4P were at least as effective in promoting learning as were those recruited under FW (holding experienced contracts constant).

Second, in terms of incentivizing effort, placed teachers working under P4P contracts elicited better performance from their students than teachers working under FW contracts (holding advertised contracts constant). Averaging over the two years of the study, the within-year effort effect of P4P was 0.11 standard deviations of pupil learning and for the second year alone, the within-year effort effect of P4P was 0.16 standard deviations. There is no evidence of a differential impact of experienced contracts by type of advertisement.

In addition to teacher characteristics and student outcomes, we observe a range of teacher behaviors. These behaviors corroborate our first finding: P4P recruits performed no worse than the FW recruits in terms of their presence, preparation, and observed pedagogy. They also indicate that the learning gains brought about by those experiencing P4P contracts may have been driven, at least in part, by improved teacher presence and pedagogy. Teacher presence was 8 percentage points higher among recruits who experienced the P4P contract compared to recruits who experienced the FW contract. This is a sizeable impact given that baseline teacher presence was close to 90 percent. And teachers who experienced P4P were more effective in their classroom practices than teachers who experienced FW by 0.10 points, as measured on a 4-point scale.

Third, on retention, teachers working under P4P contracts were no more likely to quit during the two years of the study than teachers working under FW contracts. There was also no evidence of differential selection-out on baseline teacher characteristics by experienced contract, either in terms of skills or measured motivation. On the retention margin, we therefore find little evidence to support claims made by either proponents or opponents of pay-for-performance.

To sum up, by the second year of the study, we estimate the within-year effort effect of P4P to be 0.16 standard deviations of pupil learning, with the total effect rising to 0.20 standard deviations after allowing for selection. Despite evidence of lower intrinsic motivation among those recruited under P4P, these teachers were at least as effective in promoting learning as were those recruited under FW. These results support the view that pay-for-performance can improve effort while also allaying fears of harmful effects on selection. Of course, we have studied a two-year intervention—impacts of a long-term policy might be different, particularly if P4P influences individuals’ early career decisions to train as a teacher.

3
Our findings bring new experimental results on pay-for-performance to the literature on the recruitment of civil servants in low- and middle-income countries. Existing papers have examined the impact of advertising higher _unconditional_ salaries and career-track motivations, with mixed results. In Mexico, Dal B ́o et al. (2013) find that higher base salaries attracted both skilled and motivated applicants for civil service jobs. In Uganda, Deserranno (2019) finds that the expectation of higher earnings discouraged pro-social applicants for village promoter roles, resulting in lower effort and retention. And in Zambia, Ashraf et al. (forthcoming) find that emphasis on career-track motivations for community health work, while attracting some applicants who were less pro-social, resulted in hires of equal pro-sociality and greater talent overall, leading to improvements in a range of health outcomes. By studying pay-for-performance and by separately manipulating advertised and experienced contracts, we add evidence on the compositional and effort margins of a different, and widely debated, compensation policy for civil servants.

How the teaching workforce changes in response to pay-for-performance is of interest in high-income contexts as well. In the United States, there is a large (but chiefly observational) literature on the impact of compensation on who enters and leaves the teaching workforce. Well-known studies have simulated the consequences of dismissal policies (Chetty et al., 2014b; Neal, 2011; Rothstein, 2015) or examined the role of teachers’ outside options in labor supply (Chingos and West, 2012). Recent work has examined the District of Columbia’s teacher evaluation system, where financial incentives are linked to measures of teacher performance (including student test scores): Dee and Wyckoff (2015) use a regression discontinuity design to show that low-performing teachers were more likely to quit voluntarily, while Adnot et al. (2017) confirm that these ‘quitters’ were replaced by higher-performers. In Wisconsin, a reform permitted approximately half of the state’s school districts to introduce flexible salary schemes that allow pay to vary with performance. In that setting, Biasi (2019) finds that high-value-added teachers were more likely to move to districts with flexible pay, and were less likely to quit, than their low-value-added counterparts. Our prospective, experimental study of pay-for-performance contributes to this literature methodologically but also substantively since the Rwandan labor market shares important features with high-income contexts.<sup>1</sup>

> 1Notably, there is no public sector pay premium in Rwanda, which is unusual for a low-income country and more typical of high-income countries (Finan et al., 2017). The 2017 Rwanda Labour Force Survey includes a small sample of recent Teacher Training College graduates (aged below age 30). Of these, 37 percent were in teaching jobs earning an average monthly salary of 43,431 RWF ,

4
While our paper is not the first on the broader topic of incentive-based contracts for teachers,<sup>2</sup> we go to some length to address two challenges thought to be important for policy implementation at scale. One is that the structure of the incentive should not unfairly disadvantage any particular group (Barlevy and Neal, 2012); the other is that the incentive should not be inappropriately narrow (Stecher et al., 2018). We address the first issue by using a measure of learning based on a pay-for-percentile scheme that makes student performance at all levels relevant to teacher rewards, and the second by combining this with measures of teachers’ inputs into the classroom to create a broad, composite metric. There is a small but growing literature studying pay-for-percentile schemes in education: Loyalka et al. (2019) in China, Gilligan et al. (forthcoming) in Uganda, and Mbiti et al. (2018) in Tanzania. Our contribution is to compare the effectiveness of contracts, P4P versus FW, that are based on a composite metric and are budget neutral in salary.

A final, methodological contribution of the paper, in addition to the experimental design, is the way in which we develop a pre-analysis plan. In our registered plan (AEARCTR-0002565), we pose three questions. What outcomes to study? What hypotheses to test for each outcome? And how to test each hypothesis? We answered the ‘what’ questions on the basis of theory, policy relevance and available data. With these questions settled, we then answered the ‘how’ question using blinded data. Specifically, we used a blinded dataset that allowed us to learn about a subset of the statistical properties of our data without deriving hypotheses from realized treatment responses, as advocated by, e.g., Olken (2015).<sup>3</sup> This approach achieves power gains by choosing from among specifications and test statistics on the basis of simulated power, while protecting against the risk of false positives that could arise if specifications were chosen on the basis of their realized statistical significance. The spirit of this approach is similar to recent work by Anderson and Magruder (2017) and Fafchamps and Labonne (2017).<sup>4</sup> For an experimental study in which one important dimension of variation occurs at the labor-market level, and so is potentially limited in power, the gains from these specification choices are while 15 percent were in non-teaching jobs earning a higher average monthly salary of 56,347 RWF —a _private sector_ premium of close to 30 percent.

> 2See, e.g., Imberman (2015) and Jackson et al. (2014) who provide a review.

> 3We have not found prior examples of such blinding in economics. Humphreys et al. (2013) argue for, and undertake, a related approach with partial endline data in a political science application.

> 4In contrast to those two papers, we forsake the opportunity to undertake exploratory analysis because our primary hypotheses were determined _a priori_ by theory and policy relevance. In return, we avoid having to discard part of our sample, with associated power loss.

5
particularly important. The results reported in our pre-analysis plan demonstrate that, with specifications appropriately chosen, the study design is well powered, such that even null effects would be of both policy and academic interest.

In the remainder of the paper, Sections 2 and 3 describe the study design and data, Sections 4 and 5 report and discuss the results, and Section 6 concludes.

# **2 Study design**

## **2.1 Setting**

The first tier of the study took place during the actual recruitment for civil service teaching jobs in upper primary in six districts of Rwanda in 2016.<sup>5</sup> To apply for a civil service teaching job, an individual needs to hold a Teacher Training College (TTC) degree. Eligibility is further defined by specialization. Districts solicit applications at the district-by-subject-family level, aggregating curricula subjects into three ‘families’ that correspond to the degree types issued by TTCs: math and science (TMS); modern languages (TML); and social studies (TSS). Districts invite applications between November and December, for the academic year beginning in late January/early February. Individuals keen to teach in a particular district submit one application and are then considered for all eligible teaching posts in that district in that hiring round.

Given this institutional setting, we can think of district-by-subject-family pairs as _labor markets_ . The subject-family boundaries of these labor markets are rigid; within each district, TTC degree holders are considered for jobs in pools alongside others with the same qualification. The district boundaries may be more porous, though three quarters of the new teaching jobs in our study were filled by recruits living within the district at the time of application. Since this is the majority of jobs, we proceed by treating these labor markets as distinct for our primary analysis and provide robustness checks for cross-district applications in Appendix C.<sup>6</sup> There are 18 such labor markets in our study.<sup>7</sup> This is a small number in terms of statistical power (as we address below) but not from a system-scale perspective. The study covers more than 600 hiring lines constituting over 60 percent of the country’s

> 5Upper primary refers to grades 4, 5, and 6; schools typically include grades 1 through 6.

> 6As we note in the appendix, cross-district applications would not lead us to find a selection effect where none existed but we might overstate the magnitude of any selection effect.

> 7Inference based on asymptotics could easily be invalid with 18 randomizable markets. We address this risk by committing to randomization inference for all aspects of statistical testing.

6
planned recruitment in 2016. Importantly, it is not a foregone conclusion that TTC graduates will apply for these civil service teaching jobs. Data from the 2017 Rwanda Labour Force Survey indicate that only 37 percent of recent TTC graduates were in teaching jobs, with 15 percent in non-teaching, salaried employment. This is not because the teacher labor market is tight; nationwide close to a quarter of vacancies created by a teacher leaving a school remain unfilled in the following school year (Zeitlin, 2020). A more plausible explanation is that the recent graduates in the outside sector earned a premium of close to 30 percent, making occupational choice after TTC a meaningful decision.

# **2.2 Experiment**

**Contract structure** The experiment was built around the comparison of two contracts paying a bonus on top of teacher salaries in each of the 2016 and 2017 school years, and was managed by Innovations for Poverty Action (IPA) in coordination with REB. The first of these was a P4P contract, which paid RWF 100,000 (approximately 15 percent of annual salary) to the top 20 percent of upper-primary teachers within a district, as measured by a composite performance metric.<sup>8</sup> This metric equally weighted student learning alongside three measures of teachers’ inputs into the classroom (presence, lesson preparation, and observed pedagogy). The measure of learning was based on a pay-for-percentile scheme that makes student performance at all levels relevant to teacher rewards (Barlevy and Neal, 2012).<sup>9</sup> The 2016 performance award was conditional on remaining in post during the entire 2016 school year, and was to be paid early in 2017. Likewise, the 2017 performance award was conditional on remaining in post during the entire 2017 school year, and was to be paid early in 2018. The second was a fixed-wage (FW) contract that paid RWF 20,000 to all upper-primary teachers. This bonus was paid at the same time as the performance award in the P4P contract.

Although P4P contracts based on a composite metric of teacher inputs and student performance have been used in a number of policy settings in the US (Im-

> 8The exchange rate on January 1, 2016 was 734 RWF to 1 USD, so the RWF 100,000 bonus was worth roughly 136 USD.

> 9Student learning contributed to an individual teacher’s score via percentiles within studentbased brackets so that a teacher with a particular mix of low-performing and high-performing students was, in effect, competing with other teachers with similar mixes of students. The data used to construct this measure, and the measures of teachers’ inputs, are described in Sections 3.3 and 3.4 respectively, and we explain the adaptation of the Barlevy-Neal measure of learning outcomes to a repeated cross-section of pupils in Appendix D.

7
berman, 2015; Stecher et al., 2018), such contracts have been relatively less studied in low- and middle-income countries. In their comprehensive review, Glewwe and Muralidharan (2016) discuss several evaluations of teacher incentives based on student test scores or attendance checks, but none based on a combination of both. After extensive discussions with REB about what would be suitable in this policy setting, a decision was made to use the P4P contract described above, based on a composite metric.

**Design overview** The design, summarized visually in Figure A.1, draws on a twotiered experiment, as used elsewhere (see Karlan and Zinman (2009), Ashraf et al. (2010), and Cohen and Dupas (2010) in credit-market and public-health contexts). Both tiers employ the contract variation described above.

Potential applicants, not all of whom were observed, were assigned to either advertised FW or advertised P4P contracts, depending on the labor market in which they resided. Those who actually applied, and were placed into schools, fall into one of the four groups summarized in Figure 1. For example, group _a_ denotes teachers who applied to jobs advertised as FW, and who were placed in schools assigned to FW contracts, while group _c_ denotes teachers who applied to jobs advertised as FW and who were then placed in schools re-randomized to P4P contracts. Under this experimental design, comparisons between groups _a_ and _b_ , and between groups _c_ and _d_ , allow us to learn about a pure compositional effect of P4P contracts on teacher performance, whereas comparisons along the diagonal of _a_ – _d_ are informative about the total effect of such contracts, along both margins.

Figure 1: Treatment groups among recruits placed in study schools

**First tier randomization: Advertised contracts** Our aim in the first tier was to randomize the 18 distinct labor markets to contracts, ‘treating’ all potential applicants in a given market so that we could detect the supply-side response to a particular contract. The result of the randomized assignment is that 7 of these labor

8
markets can be thought of as being in a ‘P4P only’ advertised treatment, 7 in a ‘FW only’ advertised treatment, and 4 in a ‘Mixed’ advertised treatment.<sup>10</sup> Empirically, we consider the Mixed treatment as a separate arm; we estimate a corresponding advertisement effect only as an incidental parameter.

This first-tier randomization was accompanied by an advertising campaign to increase awareness of the new posts and their associated contracts.<sup>11</sup> In November 2015, as soon as districts revealed the positions to be filled, we announced the advertised contract assignment. In addition to radio, poster, and flyer advertisements, and the presence of a person to explain the advertised contracts at District Education Offices, we also held three job fairs at TTCs to promote the interventions. These job fairs were advertised through WhatsApp networks of TTC graduates. All advertisements emphasized that the contracts were available for recruits placed in the 2016 school year and that the payments would continue into the 2017 school year. Applications were then submitted in December 2015. In January 2016, all districts held screening examinations for potential candidates. Successful candidates were placed into schools by districts during February–March 2016, and were then assigned to particular grades, subjects, and streams by their head teachers.

**Second-tier randomization: Experienced contracts** Our aim in the second tier was to randomize the schools to which REB had allocated the new posts to contracts. A school was included in the sample if it had at least one new post that was filled and assigned to an upper-primary grade.<sup>12</sup> Following a full baseline survey, schools were randomly assigned to either P4P or FW. Of the 164 schools in the second tier of the experiment, 85 were assigned to P4P and 79 were assigned to FW contracts.

> 10This randomization was performed in MATLAB by the authors. The Mixed advertised treatment arose due to logistical challenges detailed in the pre-analysis plan: the first-tier randomization was carried out at the level of the subject rather than the subject-family. An example of a districtby-subject-family assigned to the Mixed treatment is Ngoma-TML. An individual living in Ngoma with a TML qualification could have applied for an advertised Ngoma post in English on a FW contract, or an advertised Ngoma post in Kinyarwanda on a P4P contract. In contrast, KireheTML is in the P4P only treatment. So someone in Kirehe with a TML qualification could have applied for either an English or Kinyarwanda post, but both would have been on a P4P contract.

> 11Details of the promotional materials used in this campaign are provided in Appendix E.

> 12Because schools could receive multiple recruits, for different teaching specializations, it was possible for enrolled schools to contain two recruits who had experienced distinct advertised treatments. Recruits hired under the mixed advertisement treatment, and the schools in which they were placed, also met our enrollment criteria. These were similarly re-randomized to either experienced P4P or experienced FW in the second-tier randomization.

9
All upper-primary teachers—placed recruit or incumbent—within each school received the new contract. At individual applicant level, this amounted to rerandomization and hence a change to the initial assignment for some new recruits. A natural concern is that individuals who applied under one contract, but who were eventually offered another contract, might have experienced disappointment (or other negative feelings) which then had a causal impact on their behavior. To mitigate this concern, all new recruits were offered an _end-of-year retention bonus_ of RWF 80,000 on top of their school-randomized P4P or FW contract. An individual who applied under advertised P4P in the hope of receiving RWF 100,000 from the scheme, but who was subsequently re-randomized to experienced FW, was therefore still eligible to receive RWF 100,000 (RWF 20,000 from the FW contract plus RWF 80,000 as a retention bonus). Conversely, an individual who applied under advertised FW safe in the knowledge of receiving RWF 20,000 from the scheme, but who was subsequently re-randomized to experienced P4P, was still eligible for at least RWF 80,000. None of the recruits objected to the (re)randomization or turned down their re-randomized contract.

Of course, surprise effects, disappointment or otherwise, may still be present in on-the-job performance. When testing hypotheses relating to student learning, we include a secondary specification with an interaction term to allow the estimated impact of experienced P4P to differ by advertised treatment. We also explore whether surprise effects are evident in either retention or job satisfaction. We find no evidence for any surprise effect.

To ensure that teachers in P4P schools understood the new contract, we held a compulsory half-day briefing session in every P4P school to explain the intervention. This session was conducted by a team of qualified enumerators and District Education Office staff, who themselves received three days of training from the Principal Investigators in cooperation with IPA. Appendix E reproduces an extract of the English version of the enumerator manual, which was piloted before use. The sessions provided ample space for discussion and made use of practical examples. Teachers’ understanding was tested informally at the end of the session. We also held a comparable (but simpler) half-day briefing session in every FW school.

# **2.3 Hypotheses**

Pre-commitment to an analytical approach can forestall _p_ -hacking, but requires clear specification of both what to test and how to test it; this presents an opportunity, 10
as we now discuss. A theoretical model, discussed briefly below, and included in our pre-analysis plan and Appendix B, guides our choice of _what_ hypotheses to test. However, exactly _how_ to test these hypotheses in a way that maximizes statistical power is not fully determined by theory, as statistical power may depend on features of the data that could not be known in advance: the distribution of outcomes, their relationships with possible baseline predictors, and so on. We used blinded data to help decide how to test the hypotheses. In what follows we first briefly describe the theoretical model, and then discuss our statistical approach.

**Theory** The model considers a fresh graduate from teacher training who decides whether to apply for a state school teaching post, or a job in another sector (a composite ‘outside sector’). The risk neutral individual cares about compensation _w_ and effort _e_ . Her payoff is sector specific: in teaching it is _w−_ ( _e_<sup>2</sup> _−τ e_ ), while in the outside sector it is _w − e_<sup>2</sup> . The parameter _τ ≥_ 0 captures the individual’s _intrinsic motivation_ to teach, which is perfectly observed by the individual herself but not by the employer at the time of hiring.<sup>13</sup> Effort generates a performance metric _m_ = _e θ_ + _ε_ , where _θ ≥_ 1 represents her _ability_ , which is also private information at the time of hiring. Compensation corresponds to one of the four cells in Figure 1. The timing is as follows. Teacher vacancies are advertised as either P4P or FW. The individual, of type ( _τ, θ_ ), applies either to a teaching job or to an outside job. Employers hire, at random, from the set of ( _τ, θ_ ) types that apply. Thereafter, contracts are re-randomized. If the individual applies to, and is placed in a school, she learns about her experienced contract and chooses her effort level, which results in performance _m_ at the end of the year. Compensation is paid according to the experienced contract.

This model leads to the following hypotheses, as set out in our pre-analysis plan:

- I. Advertised P4P induces differential _application_ qualities;

- II. Advertised P4P affects the observable skills of recruits _placed_ in schools;

- III. Advertised P4P induces differentially intrinsically motivated recruits to be _placed_ in schools;

- IV. Advertised P4P induces the supply-side selection-in of higher- (or lower-)performing teachers, as measured by the learning outcomes of their students;

13See Delfgaauw and Dur (2007) for a related approach to modeling differential worker motivation across sectors.

11
- V. Experienced P4P creates effort incentives which contribute to higher (or lower) teacher performance, as measured by the learning outcomes of their students;

- VI. These selection and incentive effects are apparent in the composite performance metric.

The model predicts that the set of ( _τ, θ_ ) types preferring a teaching job advertised under P4P to a job in the outside sector is different from the set of types preferring a teaching job advertised under FW to a job in the outside sector. This gives Hypothesis I. Since the model abstracts from labor demand effects (by assuming employers hire at random from the set of ( _τ, θ_ ) types that apply), this prediction simply maps through to placed recruits; i.e. to Hypothesis II via _θ_ , Hypothesis III via _τ_ , and Hypotheses IV to VI via the effect of _θ_ and _τ_ on performance.<sup>14</sup> The model also predicts that any given ( _τ, θ_ ) type who applies to, and is placed in, a teaching job will exert more effort under experienced P4P than experienced FW. This gives Hypotheses V and VI via the effect of _e_ on performance.

**Analysis of blinded data** Combining several previously-known insights, we used blinded data to maximize statistical power for our main hypothesis tests.

The first insights, pertaining to simulation, are due to Humphreys et al. (2013) and Olken (2015). Researchers can use actual outcome data with the treatment variable scrambled or removed to estimate specifications in ‘mock’ data. This permits navigation of an otherwise intractable ‘analysis tree’. They can also improve statistical power by simulating treatment effects and choosing the specification that minimizes the standard error. Without true treatment assignments, the influence of any decision over eventual treatment effect estimates is unknown; thus, these benefits are garnered without risk of _p_ -hacking.<sup>15</sup>

> 14When mapping the theory to our empirical context, we distinguish between these hypotheses for two reasons: we have better data for placed recruits because we were able to administer detailed survey instruments to this well-defined sub-sample; and for placed recruits we can identify the advertised treatment effect from student learning outcomes, avoiding the use of proxies for ( _τ, θ_ ). A further consideration is that the impact of advertised treatment might differ between placed recruits and applicants due to labour-demand effects. We discuss this important issue in Section 5.

> 15This would not be true if, for example, an outcome in question was known to have different support as a function of treatment, allowing the ‘blinded’ researcher to infer treatment from the outcome variable. For our blinded pre-analysis, we only consider outcomes (TTC score, and student test scores) that are nearly continuously distributed and which we believe are likely to have the same support in all study arms. To make this analysis possible, we drew inspiration from Fafchamps and Labonne (2017), who suggest dividing labor within a research team. In our case, IPA oversaw

12
The second set of insights pertain to randomization inference. Since the marketlevel randomization in our study involves 18 randomizable units, asymptotic inference is unsuitable, so we use randomization inference. It is known that any scalar function of treatment and comparison groups is a statistic upon which a (correctlysized) randomization-inference-based test of the sharp null hypothesis could be built, but also that such statistics may vary in their statistical power in the face of any particular alternative hypothesis (Imbens and Rubin, 2015). We anticipated that, even with correctly-sized tests, the market-level portion of our design may present relatively low statistical power. Consequently, we conducted blinded analysis to choose, on the basis of statistical power, among testing approaches for several hypotheses: Hypothesis I, and a common framework for Hypotheses IV and V.<sup>16</sup> Hypothesis I is the test of whether applicants to different contracts vary in their TTC scores. Blinded analysis, in which we simulated additive treatment effects and calculated the statistical power under different approaches, suggested that ordinary least squares regression (OLS) would yield lower statistical power than would a Kolmogorov-Smirnov (KS) test of the equality of two distributions. Over a range of simulations, the KS test had between one and four times the power of OLS. We therefore committed to KS (over OLS and two other alternatives) as our primary test of this hypothesis. This prediction is borne out in Appendix Table C.1.<sup>17</sup> Hypotheses IV and V relate to the effects of advertised and experienced contracts on student test scores. Here, with the re-randomization taking place at the school level, we had many possible specifications to choose from. We examined 14 specifications (modeling random effects or fixed effects at different levels), and committed to one with the highest power. Simulations suggested that this could produce a 20 to 25 percent narrower confidence interval than in a simple benchmark specification. Comparing Table 3 to Appendix Table A.4, this was substantively borne out.<sup>18</sup> On the basis of this theory and analysis of blinded data, we settled on six primary tests: an outcome, a sample, a specification and associated test statistic, and an the data-blinding process. Results of the blinded analysis (for which IPA certified that we used only blinded data) are in our pre-analysis plan. Our RCT registry entry (AEARCTR-0002565) is accompanied by IPA’s letter specifying the date after which treatment was unblinded.

> 16Hypotheses II and III employ data that our team collected, so did not have power concerns associated with them; Hypothesis VI offered fewer degrees of freedom.

> 17The confidence interval for the KS test is roughly half the width of the corresponding OLS confidence interval: a gain in precision commensurate with more than tripling the sample size.

18Our pre-committed random effects model yields a confidence interval 86 percent as wide as that from OLS, commensurate with the power gain from increasing the sample size by a third.

13
inference procedure for each of Hypotheses I-VI, as set out in Appendix Table A.1. We also included a small number of secondary tests based on different outcomes, samples, and/or specifications. In Section 4, we report results for every primary test; secondary tests are in Section 4 or in an appendix. To aid interpretation, we also include a small amount of supplementary analysis that was not discussed in the pre-analysis plan—e.g. impacts of advertised P4P on teacher attributes beyond observable skill and intrinsic motivation, and estimates from a teacher value added model—but are cautious and make clear when this is post-hoc.

# **3 Data**

The primary analyses make use of several distinct types of data. Conceptually, these trace out the causal chain from the advertisement intervention to a sequence of outcomes: that is, from the candidates’ application decisions, to the set (and attributes) of candidates hired into schools, to the learning outcomes that they deliver, and, finally, to the teachers’ decisions to remain in the schools. In this section, we describe the administrative, survey, and assessment data available for each of these steps in the causal chain. Our understanding of these data informs our choices of specification for analysis, as discussed in detail in the pre-analysis plan.

## **3.1 Applications**

Table 1 summarizes the applications for the newly advertised jobs, submitted in January 2016, across the six districts.<sup>19</sup> Of the 2,185 applications, 1,963 come from candidates with a TTC degree—we term these _qualified_ since a TTC degree is required for the placements at stake. In the table, we present TTC scores, genders, and ages—the other observed CV characteristics—for all qualified applicants. Besides these two demographic variables, TTC scores are the only consistently measured characteristics of all applicants.

The 2,185 applications come from 1,424 unique individuals, of whom 1,194 have a TTC qualification. The majority (62 percent) of qualified applicants complete only one application, with 22 percent applying to two districts and 16 percent applying to three or more. Multiple applications are possible but not the norm, most likely because each district requires its own exam. Of those applying twice, 92 percent

> 19These data were obtained from the six district offices and represent a census of applications for the new posts across these districts.

14
Table 1: Application characteristics, by district

||Gatsibo|Kayonza|Kirehe|Ngoma|Nyagatare|Rwamagana|All|
|---|---|---|---|---|---|---|---|
|Applications|390|310|462|381|327|315|2,185|
|Qualifed|333|258|458|365|272|277|1,963|
|Has TTC score|317|233|405|338|260|163|1,716|
|Mean TTC score|0.53|0.54|0.50|0.53|0.54|0.55|0.53|
|SD TTC score|0.14|0.15|0.19|0.15|0.14|0.12|0.15|
|Qualifed female|0.53|0.47|0.45|0.50|0.44|0.45|0.48|
|Qualifed age|27.32|27.78|27.23|27.23|26.98|27.50|27.33|

applied to adjacent pairs of districts. In Appendix C, we use this geographical feature of applications to test for cross-district labor-supply effects and fail to reject the null that these are zero.

# **3.2 Teacher attributes**

During February and March 2016, we visited schools soon after they were enrolled in the study to collect baseline data using surveys and ‘lab-in-the-field’ instruments. School surveys were administered to head teachers or their deputies, and included a variety of data on management practices—not documented here—as well as administrative records of teacher attributes, including age, gender, and qualifications. The data cover all teachers in the school, regardless of their eligibility for the intervention. Teacher surveys were administered to all teachers responsible for at least one upper-primary, core-curricular subject and included questions about demographics, training, qualifications and experience, earnings, and other characteristics.

The ‘lab-in-the-field’ instruments were administered to the same set of teachers, and were intended to measure the two characteristics introduced in the theory: intrinsic motivation and ability. In the model, more intrinsically motivated teachers derive a higher benefit (or lower cost) from their efforts to promote learning. To capture this idea of other-regarding preferences towards students, taking inspiration from the work of Ashraf et al. (2014), we used a framed version of the _Dictator Game_ (Eckel and Grossman, 1996).<sup>20</sup> Teachers were given 2,000 Rwandan francs (RWF) and asked how much of this money they wished to allocate to the provision of school

> 20Previous work shows the reliability of the DG as a measure of other-regarding preferences related to intrinsic motivation (Banuri and Keefer, 2016; Brock et al., 2016; Deserranno, 2019).

15
supply packets for students in their schools, and how much they wished to keep for themselves. Each packet contained one notebook and pen and was worth 200 RWF. Teachers could decide to allocate any amount, from zero to all 2,000 RWF, which would supply ten randomly chosen students with a packet.

We also asked teachers to undertake a _Grading Task_ which measured their mastery of the curriculum in the main subject that they teach.<sup>21</sup> Teachers were asked to grade a student examination script, and had 5 minutes to determine if a series of student answers were correct or incorrect. They received a fixed payment for participation. Performance on this task was used to estimate a measure of teacher skill based on a two-parameter item response theory (IRT) model.

# **3.3 Student learning**

Student learning was measured in three rounds of assessment: baseline, the end of the 2016 school year, and the end of the 2017 school year (indexed by _r_ = 0 _,_ 1 _,_ 2). These student assessments play a dual role in our study: they provide the primary measure of learning for analysis of program impacts, and they were used in the experienced P4P arm for purposes of performance awards.

Working with the Ministry of Education, we developed comprehensive subjectand grade-specific, competency-based assessments for grades 4, 5 and 6. These assessments were based on the new Rwanda national curriculum and covered the five core subjects: Kinyarwanda, English, Mathematics, Sciences, and Social Studies. There was one assessment per grade-subject, with students at the beginning of the year being assessed on the prior year’s material. Each test aimed to cover the entire curriculum for the corresponding subject and year, with questions becoming progressively more difficult as a student advanced in the test. The questions were a combination of multiple choice and fill-in diagrams. The tests were piloted extensively; they have no ceiling effects while floor effects are limited.<sup>22</sup> In each round, we randomly sampled a subset of students from each grade to take the test. In Year 1, both baseline and endline student samples were drawn from the official school register of enrolled students compiled by the head teacher at the beginning of the year. This ensured that the sampling protocol did not create incentives for strategic

> 21See Bold et al. (2017) who use a similar approach to assess teacher content knowledge.

> 22Test scores are approximately normally distributed with a mean of close to 50 percent of questions answered correctly. A validation exercise of the test at baseline found its scores to be predictive of the national Primary Leaving Exam scores (both measured in school averages).

16
exclusion of students. In Year 2, students were assessed at the end of the year only, and were sampled from a listing that we collected in the second trimester.

Student samples were stratified by teaching _streams_ (subgroups of students taught together for all subjects). In Round 0, we sampled a minimum of 5 pupils per stream, and oversampled streams taught in at least one subject by a new recruit to fill available spaces, up to a maximum of 20 pupils per stream and 40 per grade. In rare cases of grades with more than 8 streams, we sampled 5 pupils from all streams. In Round 1, we sampled 10 pupils from each stream: 5 pupils retained from the baseline (if the stream was sampled at baseline) and 5 randomly sampled new pupils. We included the new students to alleviate concerns that teachers in P4P schools might teach (only) to previously sampled students. In Round 2, we randomly sampled 10 pupils from each stream using the listing for that year.<sup>23</sup> The tests were orally administered by trained enumerators. Students listened to an enumerator as he/she read through the instructions and test questions, prompting students to answer. The exam was timed for 50 minutes, allowing for 10 minutes per section. Enumerators administered the exam using a timed proctoring video on electronic tablets, which further ensured consistency in test administration. Individual student test results were kept confidential from teachers, parents, head teachers, and Ministry of Education officials, and have only been used for performance award and evaluation purposes in this study.

Responses were used to estimate a measure of student learning (for a given student in a given round and given subject in a given grade) based on a two-parameter IRT model. We use empirical Bayes estimates of student ability from this model as our measure of a student’s learning level in a particular grade.

# **3.4 Teacher inputs**

We collected data on several dimensions of teachers’ inputs into the classroom. This was undertaken in P4P schools only during Year 1 , and in both P4P and FW schools in Year 2. This composite metric is based on three teacher input measures (presence, lesson preparation, and observed pedagogy), and one output measure (pupil learning)—the ‘4Ps’. Here we describe the input components measured.

> 23Consequently, the number of pupils assessed in Year 2 who have also been assessed in Year 1 is limited. Because streams are reshuffled across years and because we were not able to match Year 2 pupil registers to Year 1 registers in advance of the assessment, it was not possible to sample pupils to maintain a panel across years while continuing to stratify by stream.

17
To assess the three inputs, P4P schools received three unannounced surprise visits: two spot checks during Summer 2016, and one spot check in Summer 2017. During these visits, Sector Education Officers (SEOs) from the District Education Offices (in Year 1) or IPA staff (for logistical reasons, in Year 2) observed teachers and monitored their presence, preparation and pedagogy with the aid of specially designed tools.<sup>24</sup> FW schools also received an unannounced visit in Year 2, at the same time as the P4P schools. Table A.2 shows summary statistics for each of these three input measures over the three rounds of the study.

_Presence_ is defined as the fraction of spot-check days that the teacher is present at the start of the school day. For the SEO to record a teacher present, the head teacher had to physically show the SEO that the teacher was in school.

Lesson _preparation_ is defined as the planning involved with daily lessons, and is measured through a review of teachers’ weekly lesson plans. Prior to any spot checks, teachers in grades 4, 5, and 6 in P4P schools were reminded how to fill out a lesson plan in accordance with REB guidelines. Specifically, SEOs provided teachers with a template to record their lesson preparation, focusing on three key components of a lesson—the lesson objective, the instructional activities, and the types of assessment to be used. A ‘hands-on’ session enabled teachers to practice using this template. During the SEO’s unannounced visit, he/she collected the daily lesson plans (if any had been prepared) from each teacher. Field staff subsequently used a lesson-planning scoring rubric to provide a subjective measure of quality. Because a substantial share of upper-primary teachers did not have a lesson plan on a randomly chosen audit day, we used the presence of such a lesson plan as a summary measure in both the incentivized contracts and as an outcome for analysis.

_Pedagogy_ is defined as the practices and methods that teachers use in order to impact student learning. We collaborated with both the Ministry of Education and REB to develop a monitoring instrument to measure teacher pedagogy through classroom observation. Our classroom observation instrument measured objective teacher actions and skills as an input into scoring teachers’ pedagogical performance. Our rubric was adapted from the Danielson Framework for Teaching, which is widely

> 24Training of SEOs took place over two days. Day 1 consisted of an overview of the study and its objectives and focused on how to explain the intervention (in particular the 4Ps) to teachers in P4P schools using the enumerator manual in Appendix E. During Day 2, SEOs learned how to use the teacher monitoring tools and how to conduct unannounced school visits. SEOs were shown videos recorded during pilot visits. SEOs were briefed on the importance of not informing teachers or head teachers ahead of the visits. Field staff monitored the SEOs’ adherence to protocol.

18
used in the U.S. (Danielson, 2007). The observer evaluated the teachers’ effective use of 21 different activities over the course of a full 45-minute lesson. Based on these observations and a detailed rubric, the observer provided a subjective score, on a scale from zero to three, of four components of the lesson: communication of lesson objectives, delivery of material, use of assessment, and student engagement. The teacher’s incentivized score, as well the measure of pedagogy used in our analysis, is defined as the average of these ratings across the four domains.

## **3.5 Balance**

We use the baseline data described in this section to check whether the second-tier randomization produced an appropriately ‘balanced’ experienced treatment assignment. Table 2 confirms that across a wide range of school, teacher, and student characteristics there are no statistically significant differences in means between the experienced P4P and FW treatment arms.<sup>25</sup>

# **4 Results**

Our two-tiered experiment allows us to estimate impacts of pay-for-performance on the type of individuals applying to, and being placed in, primary teaching posts (the compositional margin), and on the activities undertaken by these new recruits (the effort margin). We report these results in Sections 4.1 and 4.2 respectively. Of course, the long-run effects of pay-for-performance will depend not only on selectionin, but also selection- _out_ , as well as the dynamics of the behavioral response on the part of teachers who stay. We address dynamic issues in Section 4.3, and postpone a substantive discussion of results until Section 5. All statistical tests are conducted via randomization inference with 2,000 permutations of the experienced treatment.

## **4.1 Compositional margin of pay-for-performance**

We study three types of compositional effect of pay-for-performance. These are impacts on: the quality of applicants; the observable skill and motivation of placed recruits on arrival; and the student learning induced by these placed recruits during their first and second year on the job.

> 25Since the teacher inputs described in Section 3.4 were collected _after_ the second-tier randomization, they are not included in Table 2. See instead Appendix Table A.2.

19
Table 2: Baseline characteristics and balance of experienced P4P assignment

|Control mean<br>[St. Dev.]|Experienced P4P<br>(_p_-value)|Obs.|
|---|---|---|
|_Panel A. School attributes_|||
|Number of streams<br>9.99<br>[4.48]|-0.10<br>(0.882)|164|
|Number of teachers<br>20.47<br>[8.49]|0.56<br>(0.730)|164|
|Number of new recruits<br>1.94<br>[1.30]|0.13<br>(0.503)|164|
|Number of students<br>410.06<br>[206.71]|1.42<br>(0.985)|164|
|Share female students<br>0.58<br>[0.09]|0.00<br>(0.777)|164|
|_Panel B. Upper-primary teacher recruit att_|_ributes_||
|Female<br>0.36<br>[0.48]|-0.02<br>(0.779)|242|
|Age<br>25.82<br>[4.05]|-0.25<br>(0.616)|242|
|DG share sent<br>0.28<br>[0.33]|-0.04<br>(0.450)|242|
|Grading task score<br>-0.24<br>[0.93]|0.12<br>(0.293)|242|
|_Panel C. Pupil learning assessments_|||
|English<br>-0.00<br>[1.00]|0.04<br>(0.551)|13826|
|Kinyarwanda<br>-0.00|0.05|13831|
|[1.00]|(0.292)||
|Mathematics<br>0.00|-0.00|13826|
|[1.00]|(0.950)||
|Science<br>-0.00|0.03|13829|
|[1.00]|(0.607)||
|Social Studies<br>-0.00|0.02|13829|
|[1.00]|(0.670)||

Notes: The table provides summary statistics for attributes of schools, teachers (new recruits placed in upper primary only), and students collected at baseline. The first column presents means in FW schools, (with standard deviations in brackets); the second column presents estimated differences between FW and P4P schools (with randomization inference _p_ -values in parentheses). The sample in Panel B consists of new recruits placed in upper-primary classrooms at baseline, who undertook the lab-in-the-field exercises. In Panel B, Grading Task IRT scores are standardized based on the distribution among incumbent teachers. In Panel C, student learning IRT scores are standardized based on the distribution in the experienced FW arm.

20
<!-- Start of picture text --> 1 ae<br>ear<br>0.9; |————__ papFW ‘A2<br>Mixed<br>08<br>i<br>rat0 07 }|<br>° 06 Fif<br>805 }<br>uw 04 #|<br>y,<br>01 a<br>es we<br>p bate<br>0 0.2 0.4 0.6 0.8 1<br>TTC scores<br><!-- End of picture text -->
and advertised FW. The KS test-statistic has a value of 0.026, with a _p_ -value of 0.909. Randomization inference is well-powered, meaning that we can rule out even small effects on the TTC score distribution: a 95 percent confidence interval based on inversion of the randomization inference test rules out additive treatment effects outside of the range [ _−_ 0 _._ 020 _,_ 0 _._ 020]. (The OLS estimate of this additive treatment effect is -0.001, as reported in Table C.1.) We therefore conclude that there was no meaningful impact of advertised P4P on the TTC final exam scores of applicants.<sup>26</sup> Below, we move on to consider impacts of advertised P4P on the quality of applicants who were offered a post and chose to accept it—a subset that we term _placed recruits_ . It is worth emphasizing that we may find results here even though there is no evidence of an impact on the distribution of TTC score of applicants. This is because, for this well-defined set of placed recruits, we have access to far richer data: lab-in-the-field instruments measuring attributes on arrival, as well as measures of student learning in the first and second years on the job.

**Skill and motivation of placed recruits** Along the lines suggested by Dal B ́o and Finan (2016), we explore whether institutions can attract the most capable or the most intrinsically motivated into public service. We include multidimensional skill and motivation types in the theoretical model and test the resulting hypotheses (Hypotheses II and III) using the data described in Section 3.2. Specifically, we use the Grading Task IRT score to measure a placed recruit’s skill on arrival, and the framed Dictator Game share sent to capture baseline intrinsic motivation.

Our primary tests use these baseline attributes of placed recruits as outcomes. For attribute _x_ of teacher _j_ with qualification _q_ in district _d_ , we estimate a regression of the form where treatment _Tqd_<sup>_A_denotesthecontractualconditionunderwhichacandidate</sup> applied.<sup>27</sup> Our test of the null hypothesis is the _t_ statistic associated with coefficient _τA_ . We obtain a randomization distribution for this _t_ statistic under the sharp null of no effects for any hire by estimating equation (2) under the set of feasible

> 26This conclusion is further substantiated by the battery of secondary tests in Appendix C.

> 27Here and throughout the empirical specifications, we will define _Tqd A_<sup>asa</sup><sup>_vector_thatincludes</sup> indicators for both the P4P and mixed-treatment advertisement condition. However, for hypothesis testing, we are interested only in the coefficient on the pure P4P treatment. Defining treatment in this way ensures that only candidates who applied (and were placed) under the pure FW treatment are considered as the omitted category here, to which P4P recruits will be compared.

22
<!-- Start of picture text --> 1 oe<br>09 Pap Ai<br>08 Mixed pe<br>; f<br>07 hf<br>Le0 oFJY<br>°= 06 ae)J<br>805 ifa<br>§ 04 /<br>03 dS.VA<br>LE<br>02 Te<br>04 p<br>wet<br>0 ae<br>3 2 A 0 1 2<br>Teacher ability<br><!-- End of picture text --> <!-- Start of picture text --> 41 ie<br>09 Pap ee _e’<br>Mixed a al<br>08 ee a“<br>07 a a<br>uw 7Pe rae Le<br>°O06 Ya ue a<br>=805 f “ 7<br>B04 |<br>03 “<br>02<br>04<br>0<br>0 0.2 04 0.6 08 1<br>Teacher DG contribution<br><!-- End of picture text -->
further _exploratory_ analysis of the impact of advertised P4P. Specifically, we estimate regressions of the form given in equation (2) for four additional teacher attributes: age, gender, risk aversion, and an index capturing the Big Five personality traits.<sup>28</sup> Results are reported in Table A.3, with details of the variable construction provided in the table-note. We are unable to reject the sharp null of no advertised P4P treatment effect for any of these exploratory outcomes.

**Student learning induced by placed recruits** The skill and motivation of placed recruits on arrival are policy relevant insofar as these attributes translate into teacher effectiveness. To assess this, we combine experimental variation in the advertised contracts to which recruits applied, with the second-stage randomization in experienced contracts under which they worked. This allows us to estimate the impact of advertised P4P on the student learning induced by these recruits, holding constant the experienced contract—a pure compositional effect (Hypothesis IV).

Our primary test is derived from estimates on student-subject-year level data. The advertised treatment about which a given student’s performance is informative depends on the identity of the placed recruit teaching that particular subject via qualification type and district. We denote this by _Tqd_<sup>_A_for teacher</sup><sup>_j_with qualification</sup> type _q_ in district _d_ , and suppress the dependence of the teacher’s qualification _q_ on the subject _b_ , stream _k_ , school _s_ , and round _r_ , which implies that _q_ = _q_ ( _bksr_ ). The experienced treatment is assigned at the school level, and is denoted by _Ts_<sup>_E_.We</sup> pool data across the two years of intervention to estimate a specification of the type for the learning outcome of student _i_ in subject _b_ , stream _k_ , school _s_ , and round _r_ . We define _j_ = _j_ ( _bksr_ ) as an identifier for the teacher assigned to that subjectstream-school-round. The variable _Ij_ is an indicator for whether the teacher is an incumbent, and the index _q_ = _q_ ( _j_ ) denotes the qualification type of teacher _j_ if that teacher is a recruit (and is undefined if the teacher is an incumbent, so that _Tqd_<sup>_A_is</sup> always zero for incumbents). Drawing on the pseudo-panel of student outcomes, the variable ̄ _zks,r−_ 1 denotes the vector of average outcomes in the once-lagged assessment among students placed in that stream, and its coefficient, _ρbr_ , is subject- and round-

> 28Here we follow Dal B ́o et al. (2013) who measure the risk preferences and Big Five personality traits of applicants for civil service jobs in Mexico, and Callen et al. (2018) who study the relevance of Big Five personality traits for the performance of health workers in Pakistan.

24
specific. The coefficient of interest is _τA_ : the average of the within-year effect of advertised P4P on pupil learning in Year 1 and the within-year effect of advertised P4P on pupil learning in Year 2.<sup>29</sup> The theoretical model of Appendix B, as well as empirical evidence from other contractual settings (Einav et al., 2013), suggests that pay-for-performance may induce selection on the _responsiveness_ to performance incentives. If so, then the impact of advertised treatment will depend on the contractual environment into which recruits are placed. Consequently, we also estimate a specification that allows advertised treatment effects to differ by experienced treatment, including an interaction term between the two treatments. This interacted model takes the form Here, the compositional effect of advertised P4P among recruits placed in FW schools is given by _τA_ (a comparison of on-the-job performance across groups _a_ and _b_ , as defined in Figure 1). Likewise, the compositional effect of advertised P4P among recruits placed in P4P schools is given by _τA_ + _τAE_ (a comparison of groups _c_ and _d_ ). If _τAE_ is not zero, then this interacted model yields the more policy relevant estimands (Muralidharan et al., 2019). Noting the distinction between estimands and test statistics (Imbens and Rubin, 2015), we pre-specified the pooled coefficient _τA_ from equation (3) as the primary test statistic for the presence of compositional effects. Our simulations, using blinded data, show that this pooled test is better powered under circumstances where the interaction term, _τAE_ , is small.

We estimate equations (3) and (4) by a linear mixed effects model, allowing for normally distributed random effects at the student-round level.<sup>30</sup> Randomization inference is used throughout. To do so, we focus on the distribution of the estimated _z_ -statistic (i.e., the coefficient divided by its estimated standard error), which allows rejections of the sharp null of no effect on any student’s performance to be interpreted, asymptotically, as rejection of the non-sharp null that the coefficient

> 29We focus on _within-year_ impacts because there is not a well-defined cumulative treatment effect. Individual students receive differing degrees of exposure to the advertised treatments depending on their path through streams (and hence teachers) over Years 1 and 2.

> 30In our pre-analysis plan, simulations using the blinded data indicated that the linear mixed effects model with a student-round normal random effects would maximize statistical power. We found precisely this in the unblinded data. For completeness, and purely as supplementary analysis, we also present estimates and hypotheses tests via ordinary least squares. See Appendix Table A.4. These OLS estimates are generally larger in magnitude and stronger in statistical significance.

25
is equal to zero (DiCiccio and Romano, 2017). Inference for _τA_ is undertaken by permutation of the advertised treatment, _T_<sup>_A_</sup> _∈T_<sup>_A_</sup> , while inference for _τE_ likewise proceeds by permuting the experienced treatment _T_<sup>_E_</sup> _∈T_<sup>_E_</sup> . To conduct inference about the interaction term, _τAE_ in equation (4), we simultaneously permute both dimensions of the treatment, considering pairs ( _T_<sup>_A_</sup> _, T_<sup>_E_</sup> ) from the set _T_<sup>_A_</sup> _× T_<sup>_E_</sup> .

Table 3: Impacts on student learning, linear mixed effects model

||Pooled|Year 1|Year 2|
|---|---|---|---|
|_Model A: Direct efects o_|_nly_|||
|Advertised P4P (_τA_)|0.01<br>[-0.04, 0.08]<br>(0.75)|-0.03<br>[-0.06, 0.03]<br>(0.20)|0.04<br>[-0.05, 0.16]<br>(0.31)|
|Experienced P4P (_τE_)|0.11<br>[0.02, 0.21]<br>(0.02)|0.06<br>[-0.03, 0.15]<br>(0.17)|0.16<br>[0.04, 0.29]<br>(0.00)|
|Experienced P4P _×_|-0.06|-0.05|-0.09|
|Incumbent (_λE_)|[-0.20, 0.08]<br>(0.36)|[-0.19, 0.11]<br>(0.54 )|[-0.24, 0.07]<br>(0.27 )|
|_Model B: Interactions bet_|_ween advertised _|_and experienc_|_ed contracts_|
|Advertised P4P (_τA_)|0.01<br>[-0.05, 0.14]<br>(0.46)|-0.02<br>[-0.06, 0.07]<br>(0.62)|0.03<br>[-0.05, 0.21]<br>(0.22)|
|Experienced P4P (_τE_)|0.12<br>[0.05, 0.25]<br>(0.01)|0.06<br>[-0.01, 0.19]<br>(0.10)|0.18<br>[0.08, 0.33]<br>(0.00)|
|Advertised P4P _×_|-0.03|-0.01|-0.04|
|Experienced P4P (_τAE_)|[-0.17, 0.09]<br>(0.51)|[-0.15, 0.10]<br>(0.65)|[-0.22, 0.13]<br>(0.58)|
|Experienced P4P _×_|-0.08|-0.05|-0.11|
|Incumbent (_λE_)|[-0.31, 0.15]<br>(0.43)|[-0.30, 0.18]<br>(0.56)|[-0.36, 0.14]<br>(0.38)|
|Observations|154594|70821|83773|

Notes: For each estimated parameter, or combination of parameters, the table reports the point estimate (stated in standard deviations of student learning), 95 percent confidence interval in brackets, and _p_ -value in parentheses. Randomization inference is conducted on the associated _z_ statistic. The measure of student learning is based on the empirical Bayes estimate of student ability from a two-parameter IRT model, as described in Section 3.3.

Results are presented in Table 3. Pooling across years, the compositional effect of 26
<!-- Start of picture text --> 1 yop<br>0.9 ---- FW f{----<br>Pap wy<br>0.8 f ii<br>0.7 ve<br>iS}2= 06 {?Aa<br>5 0.5 }}<br>ie /<br>a£04 {4f<br>0.3 fh<br>0.22 S|aJ)<br>0.1 fojo}<br>ol== --'<br>-0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8<br>‘Teacher value added, Year 1<br><!-- End of picture text -->

<!-- Start of picture text --> 1 poo<br>---- FW / _—<br>0.9 pap If<br>0.8 1}7¢ f<br>0.7 if<br>2oO= 0.6 ¢)i|f<br>5 05 yt<br>E<br>a&£04 i )<br>0.3 }<br>0.2 Jj<br>0.1 . A?<br>0 A<br>-1 -0.5 0 0.5 1 1.5 2<br>Teacher value added, Year 2<br><!-- End of picture text -->
76th percentile in the distribution of (empirical Bayes estimates of) teacher value added for placed recruits who applied under FW. The TVA model also reveals the impact of advertised P4P on the distribution of teacher effectiveness. Figure 4b shows that the distribution of teacher value added among recruits in their second year on the job is better, by first order stochastic dominance, under advertised P4P than advertised FW. This finding is consistent with the view that a contract that rewards the top quintile of teachers attracts individuals who deliver greater learning.

# **4.2 Effort margin of pay-for-performance**

Having studied the type of individuals applying to, and being placed in, upperprimary posts, we now consider the activities undertaken by these new recruits.

**Student learning induced by placed recruits** We start by using the two-tiered experimental variation to estimate the impact of experienced P4P on the student learning induced by the placed recruits, holding constant the advertised contract—a pure effort effect (Hypothesis V). Our primary test uses the specification in equation (3), again estimated by a linear mixed effects model. The coefficient of interest is now _τE_ . To investigate possible ‘surprise effects’ from the re-randomization, we also consider the interacted specification of equation (4). In this model, _τE_ gives the effect of experienced P4P among recruits who applied under FW contractual conditions (a comparison of groups a and c, as defined in Figure 1), while _τE_ + _τAE_ gives the effect of experienced P4P among recruits who applied under P4P contractual conditions (a comparison of groups b and d). If recruits are disappointed, because it is groups b and c who received the surprise, _τE_ should be smaller than _τE_ + _τAE_ .<sup>31</sup>

Results are presented in Table 3. Pooling across years, the within-year effect of experienced P4P is 0 _._ 11 standard deviations of pupil learning (Model A, second row). The randomization inference _p_ -value is 0 _._ 03, implying that we can reject the sharp null of no experienced P4P treatment effect on placed recruits at the 5 percent level. We do not find evidence of disappointment caused by the re-randomization. The interaction term is insignificant (Model B, third row) and, in point-estimate terms, _τE_ is larger than _τE_ + _τAE_ . As was the case for the compositional margin,

> 31We are grateful to a referee for highlighting a further interpretation: _τE_ in Model B is the policy-relevant estimate of experienced P4P at the start of any unexpected transition to P4P, while _τE_ + _τAE_ is the policy-relevant estimate for that effect slightly further into a transition—the effect of P4P on a cohort anticipating P4P.

28
the effort effect of experienced P4P on student learning appears to strengthen over time. By the second year of the study, the within-year effort effect of P4P was 0 _._ 16 standard deviations of pupil learning.<sup>32</sup> To put this in perspective, we compare the magnitude of this effort effect to impacts in similar studies in the US, and beyond. Sojourner et al. (2014) study pay-for-performance schemes in Minnesota, typically based on a composite metric of subjective teacher evaluation and student performance, and find an effect of 0.03 standard deviations of pupil learning. Dee and Wyckoff (2015) study a high-stakes incentive over a composite metric in Washington, DC, and find effects consistent with those of Sojourner et al. (2014) in terms of the implied magnitude of effects on pupil learning. Glewwe and Muralidharan (2016) review a range of studies, including several in Benin, China, India and Kenya that employ incentives for either students or teachers based solely on student performance; effect sizes are larger, typically above 0.2 standard deviations of pupil learning. Our effort effect falls within this range and is of a comparable magnitude to the impact in Duflo et al. (2012), who study incentives for teacher attendance in India.

**Dimensions of the composite performance metric** The results in Table 3 speak to the obvious policy question, namely whether there are impacts of advertised and experienced P4P contracts on student learning. For completeness, and to gain an understanding into mechanisms, we complete our analysis by studying whether there are impacts on the _contracted_ metrics which are calculated at teacher-level (Hypothesis VI). For these tests, we use the following specifications:

_mjqsdr_ = _τATqd_<sup>_A_+</sup><sup>_τET E_</sup> _s_<sup>+</sup><sup>_τAET A_</sup> _qd_<sup>_T E_</sup> _s_<sup>+</sup><sup>_λIIj_+</sup><sup>_λET E_</sup> _s_<sup>_Ij_+</sup><sup>_γq_+</sup><sup>_δd_+</sup><sup>_ψr_+</sup><sup>_ejqsdr,_(6)</sup> for the metric of teacher _j_ with qualification _q_ in school _s_ of district _d_ , as observed in post-treatment round _r_ . As above, the variable _Ij_ is an indicator for whether the teacher is an incumbent (recall that _Tqd_<sup>_A_isalwayszeroforincumbents).33Alinear</sup> mixed effects model with student-level random effects is no longer applicable; out32Across all specifications, the interaction term between experienced P4P and an indicator for incumbent teachers is negative, though statistically insignificant, and smaller in magnitude than the direct effect of experienced P4P, implying a weaker—though still positive—effect of P4P on incumbents in point-estimate terms.

> 33Note that any attribute of recruits themselves, even if observed at baseline, suffers from the ‘bad controls’ problem, as the observed values of this covariate could be an outcome of the advertised

29
comes are constructed at the teacher-level, and given their rank-based construction, normality does not seem a helpful approximation to the distribution of error terms. As stated in our pre-analysis plan, we therefore estimate equations (5) and (6) with a round-school random-effects estimator to improve efficiency. The permutations of treatments used for inferential purposes mirror those above.

Results are reported in Table 4 and, to the extent available, are based on pooled data.<sup>34</sup> Consistent with the pooled results in Table 3, we see a positive and significant impact of experienced P4P on both the summary metric and the learning sub-component. The specifications with teacher inputs as dependent variables suggest that this impact on student learning is driven, at least in part, by improvements in teacher presence and pedagogy. Teacher presence was 8 percentage points higher among recruits who experienced the P4P contract compared to recruits who experienced the FW contract; an impact that is statistically significant at the 1 percent level and sizeable in economic terms given that baseline teacher presence was already 90 percent. Recruits who experienced P4P were 0.10 points more effective in their classroom practices than recruits receiving FW, although this impact is weaker in terms of statistical significance (and it is possible that this improvement occurred only during the observation). We find no evidence of impacts on lesson planning.

# **4.3 Dynamic effects**

Our two-tiered experiment was designed to evaluate the impact of pay-for-performance and, in particular, to quantify the relative importance of a compositional margin at the recruitment stage versus an effort margin on the job. The hypotheses specified in our pre-analysis plan refer to selection-in and incentives among placed recruits. Since within-year teacher turnover was limited by design and within-year changes in teacher skill and motivation are likely small, the total effect of P4P in Year 1 can plausibly only be driven by a change in the type of teachers recruited and/or a change in effort resulting from the provision of extrinsic incentives.

Interpreting the total effect of P4P in Year 2 is more complex, however. First, we made no attempt to discourage _between_ -year teacher turnover, and so there is the possibility of a further compositional margin at the retention stage (c.f. Muralidharan and Sundararaman 2011). Experienced P4P may have selected-out the low treatment. These variables are therefore not included as independent variables.

> 34As discussed in Section 3.4, FW schools only received unannounced visits to measure teacher inputs in Year 2.

30
Table 4: Estimated effects on dimensions of the composite 4P performance metric

||Summary<br>metric|Preparation|Presence|Pedagogy|Pupil<br>learning|
|---|---|---|---|---|---|
|_Model A: Direct efects on_|_ly_|||||
|Advertised P4P (_τA_)|-0.04<br>[-0.09, 0.01]<br>(0.11)|0.07<br>[-0.13, 0.32]<br>(0.40)|0.00<br>[-0.05, 0.07]<br>(0.93)|0.03<br>[-0.06, 0.10]<br>(0.42)|-0.02<br>[-0.08, 0.02]<br>(0.27)|
|Experienced P4P (_τE_)|0.23|0.02|0.08|0.10|0.09|
||[0.19, 0.28]<br>(0.00)|[-0.13, 0.16]<br>(0.84)|[0.02, 0.14]<br>(0.01)|[-0.00, 0.21]<br>(0.05)|[0.03, 0.15]<br>(0.00)|
|Experienced P4P _×_|0.03|0.07|-0.01|0.07|-0.00|
|Incumbent (_λE_)|[-0.01, 0.07]<br>(0.10)|[-0.03, 0.18]<br>(0.17)|[-0.06, 0.05]<br>(0.70)|[-0.01, 0.16]<br>(0.11)|[-0.04, 0.03]<br>(0.86)|
|_Model B: Interactions bet_|_ween advertised_|_and experienc_|_ed contracts_|||
|Advertised P4P (_τA_)|-0.03<br>[-0.12, 0.05]<br>(0.42)|0.16<br>[-0.11, 0.48]<br>(0.19)|-0.01<br>[-0.16, 0.17]<br>(0.86)|0.12<br>[-0.27, 0.55]<br>(0.44)|-0.01<br>[-0.12, 0.11]<br>(0.91)|
|Experienced P4P (_τE_)|0.22<br>[0.15, 0.29]<br>(0.00)|-0.00<br>[-0.26, 0.25]<br>(0.97)|0.08<br>[-0.01, 0.16]<br>(0.07)|0.17<br>[-0.05, 0.38]<br>(0.12)|0.08<br>[0.00, 0.16]<br>(0.04)|
|Advertised P4P _×_|-0.02|-0.11|0.02|-0.11|-0.03|
|Experienced P4P (_τAE_)|[-0.11, 0.07]<br>(0.65)|[-0.45, 0.23]<br>(0.53)|[-0.12, 0.16]<br>(0.69)|[-0.45, 0.24]<br>(0.53)|[-0.15, 0.08]<br>(0.64)|
|Experienced P4P _×_|0.05|0.09|-0.01|0.00|0.00|
|Incumbent (_λE_)|[-0.01, 0.10]<br>(0.07)|[-0.07, 0.26]<br>(0.27)|[-0.09, 0.07]<br>(0.82)|[-0.13, 0.14]<br>(0.96)|[-0.05, 0.06]<br>(0.90)|
|Observations|3996|2514|3455|2136|3049|
|FW recruit mean|0.49|0.65|0.89|1.98|0.48|
|(SD)|(0.22)|(0.49)|(0.31)|(0.57)|(0.27)|
|FW incumbent mean|0.37|0.50|0.87|2.05|0.45|
|(SD)|(0.24)|(0.50)|(0.33)|(0.49)|(0.28)|

Notes: For each estimated parameter, the table reports the point estimate, 95 percent confidence interval in brackets, and _p_ -value (or for FW means, standard deviations) in parentheses. Randomization inference is conducted on the associated _t_ statistic. All estimates are pooled across years, but outcomes are observed in the FW arm during the second year only. Outcomes are constructed at teacher-round-level as follows: _preparation_ is a binary indicator for existence of a lesson plan on a randomly chosen spot-check day; _presence_ is the fraction of spot-check days present at the start of the school day; _pedagogy_ is the classroom observation score, measured on a four-point scale; and _pupil learning_ is the Barlevy-Neal percentile rank. The _summary metric_ places 50 percent weight on learning and 50 percent on teacher inputs, and is measured in percentile ranks.

31
skilled (Lazear, 2000) or, more pessimistically, the highly intrinsically motivated. Second, given the longer time frame, teacher characteristics could have changed. Experienced P4P may have eroded a given teacher’s intrinsic motivation (as hypothesized in the largely theoretical literature on motivational crowding out) or, more optimistically, encouraged a given teacher to improve her classroom skills. In this section, we conduct an exploratory analysis of these dynamic effects.<sup>35</sup>

**Retention effects** We begin by exploring whether experienced P4P affects retention rates among recruits. Specifically, we look for an impact on the likelihood that a recruit is still employed at midline in February 2017 at the start of the Year 2; i.e. after experiencing pay-for-performance in Year 1, although before the performance awards were announced. To do so, we use a linear probability model of the form

where _employediqd_ 2 is an indicator for whether teacher _i_ with subject-family qualification _q_ in district _d_ is still employed by the school at the start of Year 2, and _γq_ and _δd_ are the usual subject-family qualification and district indicators.

As the first column of Table 5 reports, our estimate of _τE_ is zero with a randomization inference _p_ -value of 0.96. There is no statistically significant impact of experienced P4P on retention of recruits; the retention rate is practically identical—at around 80 percent—among recruits experiencing P4P and those experiencing FW.

It is worth noting that there is also no impact of experienced P4P on _intentions_ to leave in Year 3. In the endline survey in November 2017, we asked teachers the question:“How likely is it that you will leave your job at this school over the coming year?”. Answers were given on a 5-point scale. For analytical purposes we collapse these answers into a binary indicator coded to 1 for ‘very likely’ or ‘likely’ and 0 otherwise, and estimate specifications analogous to equations (5) and (6). As the second column of Table A.5 shows, there is no statistically significant impact of experienced P4P on recruits’ self-reported likelihood of leaving in Year 3. Our estimate of _τE_ is _−_ 0 _._ 06 with a randomization inference _p_ -value of 0.40.

Of course, a retention rate of 79 percent implies 21 percent attrition from Year 1

> 35We emphasize that this material is exploratory; the hypotheses tested in this section were not part of our pre-analysis plan. That said, the structure of the analysis in this section does follow a related pre-analysis plan (intended for a companion paper) which we uploaded to our trial registry on October 3, 2018, _prior_ to unblinding of our data.

32
Table 5: Retention of placed recruits

||(1)|(2)||(3)|
|---|---|---|---|---|
|Experienced P4P|0.00<br>[0.96]|-0.04<br>[0.41]||-0.08<br>[0.23]|
|Interaction||-0.05||0.16|
|||[0.38]||[0.36]|
|Heterogeneity by. . .||Grading Task|Dictator|Game|
|Observations|249|238||238|

Notes: For each estimated parameter, the table reports the point estimate and _p_ -value in brackets. Randomization inference is conducted on the associated _t_ statistic. In each column, the outcome is an indicator for whether the teacher is still employed at the start of Year 2. The mean of this dependent variable for FW recruits is 0.79. In the second column, the specification includes an interaction of experienced treatment with the teacher’s baseline Grading Task IRT score (not demeaned); in the third column, the interaction is with the teacher’s share sent in the baseline framed Dictator Game (again not de-meaned). All specifications include controls for districts and subjects of teacher qualification.

to Year 2, which is non-negligible. And the fact that retention _rates_ are similar does not rule out the possibility of an impact of experienced P4P on the _type_ of recruits retained. To explore this, we test whether experienced P4P induces differentially skilled recruits to be retained. Here, we use teachers’ performance on the baseline Grading Task in the primary subject they teach to obtain an IRT estimate of their ability in this subject, denoted _zi_ , and estimate an interacted model of the form Inference for the key parameter, _ζ_ , is undertaken by performing randomization inference for alternative assignments of the school-level experienced treatment indicator. As the second column of Table 5 reports, our estimate of _ζ_ is _−_ 0 _._ 05, with a randomization inference _p_ -value of 0 _._ 38. There is not a significant difference in selection-out on baseline teacher skill across the experienced treatments. Hence, there is no evidence that experienced P4P induces differentially skilled recruits to be retained.

We also test whether experienced P4P induces differentially intrinsically motivated recruits to be retained. Here, we use the contribution sent in the framed Dictator Game played by all recruits at baseline, denoted _xi_ , and re-estimate the interacted model in equation (8), replacing _zi_ with _xi_ . As the third column of 33
Table 5 reports, our estimate of _ζ_ in this specification is 0 _._ 16, with a randomization inference _p_ -value of 0 _._ 36. There is not a significant difference in selection-out on baseline teacher intrinsic motivation across the experienced treatments. Hence, there is also no evidence that experienced P4P induces differentially intrinsically motivated recruits to be retained.

**Changes in retained teacher characteristics** To assess whether experienced P4P changes within-retained-recruit teacher skill or intrinsic motivation from baseline to endline, we estimate the following ANCOVA specification

where _yiqsd_ 2 is the characteristic (raw Grading Task score or framed Dictator Game contribution) of retained recruit _i_ with qualification _q_ in school _s_ and district _d_ at endline (round 2), and _yiqsd_ 0 is this characteristic of retained recruit _i_ at baseline (round 0). As the first column of Table 6 reports, our estimate of _τE_ in the Grading Task specification is 0 _._ 57, with a randomization inference _p_ -value of 0.63. Our estimate of _τE_ in the Dictator Game specification is _−_ 0 _._ 04, with a randomization inference _p_ -value of 0.06. Both estimates are small in magnitude and, in the case of the Dictator Game share sent, we reject the sharp null only at the 10 percent level. Hence, to the extent that contributions in the Dictator Game are positively associated with teachers’ intrinsic motivation, we find no evidence that the _rising_ effects of experienced P4P from Year 1 to Year 2 are driven by _positive_ changes in our measures of within-retained-recruit teacher skill or intrinsic motivation.<sup>36</sup> Before moving on, it is worth noting that the Dictator Game result could be interpreted as weak evidence that the experience of P4P contracts crowded out the intrinsic motivation of recruits. We do not have any related measures observed at both baseline and endline with which to further probe _changes_ in motivation. However, we do have a range of related measures at endline: job satisfaction, likelihood of leaving, and positive/negative affect.<sup>37</sup> As Table A.5 shows, there is no statistically significant impact of experienced P4P on any of these measures.

Further substantiating this point, Table A.6 shows the distribution of answers

> 36Although repeated play of lab experimental games may complicate interpretation in some contexts, several factors allay this concern here. First, unlike strategic games, the ‘Dictator Game’ has no second ‘player’ about whom to learn. Second, the two rounds of play were fully two years apart.

> 37We follow Bloom et al. (2015) in using the Maslach Burnout Index to capture job satisfaction and the Clark-Tellgen Index of positive and negative affect to capture the overall attitude of teachers.

34
Table 6: Characteristics of retained recruits at endline

||Grading|Task|Dictator|Game|
|---|---|---|---|---|
|Experienced P4P||0.57||-0.04|
|||[0.63]||[0.06]|
|Observations||170||169|

Notes: For each estimated parameter, the table reports the point estimate and _p_ -value in brackets. Randomization inference is conducted on the associated _t_ statistic. In the first column, the outcome is the Grading Task score of the teacher at endline on a (raw) scale from 0 to 30; in the second column, it is the teacher’s share sent in the framed Dictator Game played at endline. All specifications include the outcome measured at baseline and controls for district and subject-of-qualification.

to the endline survey question: “What is your overall opinion about the idea of providing high-performing teachers with bonus payments on the basis of objective measures of student performance improvement?”<sup>38</sup> The proportion giving a favorable answer exceeds 75 percent in every study arm. In terms of Figure 1, group _a_ (recruits who both applied for and experienced FW) had the most negative view of pay-for-performance, while group _c_ (who applied for FW but experienced P4P) had the most positive view. Hence it seems that it was the idea, rather than the reality, of pay-for-performance that was unpopular with (a minority of) recruits.<sup>39</sup>

# **5 Discussion**

**Compositional margin** To recap from Section 4.1, we find no evidence of an advertised treatment impact on the measured quality of applicants for upper-primary teaching posts in study districts, but we do find evidence of an advertised treatment impact on the measured intrinsic motivation of individuals who are placed into study schools. We draw three conclusions from these results.

First, potential applicants were aware of, and responded to, the labor market intervention. The differences in distributions across advertised treatment arms in Figure 3b (Dictator Game share sent) and Figure 4b (teacher valued added in Year 2) show that the intervention changed behavior. Since these differences are for placed recruits not applicants, it could be that this behavior change was on the labor

> 38We follow the phrasing used in the surveys run by Muralidharan and Sundararaman (2011a).

> 39Consistent with our failure to find ‘surprise effects’ in student learning, there is no evidence that the re-randomization resulted in hostility toward pay-for-performance; if anything the reverse.

35
demand rather than supply side. In Appendix Figure A.2 we plot the empirical probability of hiring as a quadratic function of the rank of an applicant’s TTC score within the set of applicants in their district. It is clear from the figure that the predicted probabilities are similar across P4P and FW labor markets. We also test formally whether the probability of hiring, as a function of CV characteristics (TTC score, age and gender), is the same under both P4P and FW advertisements.<sup>40</sup> We find no statistically significantly differences across advertised treatment arms.

Second, the supply-side response was, if anything, beneficial for student learning. The P4P contract negatively selected-in the attribute measured by the baseline Dictator Game. However, Appendix Table D.1 shows that the rank correlation between the baseline DG share sent by recruits and their teacher value added is small and not statistically significant. Consistent with this, our primary test rules out meaningful negative effects of advertised P4P on student learning. In fact, our supplementary analyses—the OLS estimates in Appendix Table A.4 and the distributions of teacher value added in Figure 4—point to _positive_ effects on learning by recruits’ second year on the job. It therefore appears that only positively selected attribute(s) mattered, at least in the five core subjects that we assessed.

Finally, districts would struggle to achieve this compositional effect directly via the hiring process. The positively selected attribute(s) were not evident in the metrics observed at baseline—either in TTC scores, or in the Grading Task scores that districts could in principle adopt.<sup>41</sup> This suggests that there is not an obvious demand-side policy alternative to contractually induced supply-side selection.

**Effort margin** To recap from Section 4.2, we find evidence of a positive impact of experienced P4P on student learning, which is considerably larger (almost tripling in magnitude) in recruits’ second year on the job. In light of Section 4.3, we draw the following conclusions from these results.

The additional learning achieved by recruits working under P4P, relative to re-

> 40Note that this is a sufficient but not necessary test of the absence of a demand-side response. It is sufficient because districts do not interview applicants, so CVs give us the full set of characteristics that could determine hiring. It is not necessary, however, because we observe hires rather than offers. The probability that an offer is accepted could be affected by the advertised contract associated with that post, even if applicants apply to jobs of both types and even if DEOs do not take contract offer types into account when selecting the individuals to whom they would like to make offers.

> 41An alternative explanation for the null KS test on applicant TTC scores is that individuals applied everywhere. If this were true, we would expect to see most candidates make multiple applications, and a rejection of the null in a KS test on _placed recruits’_ TTC scores (if the supplyside response occurred at acceptance rather than application). We do not see either in the data.

36
cruits working under FW, is unlikely to be due to selection-out—the compositional margin famously highlighted by Lazear (2000). Within-year teacher turnover was limited by design. Between-year turnover did happen but cannot explain the experienced P4P effect. In Appendix D, we show that the rank correlation between recruits’ baseline Grading Task IRT score and their teacher value added is positive. However, in Section 4.3 we reported that, if anything, selection-out on baseline teacher skill runs the wrong way to explain the experienced P4P effect.

Neither is the experienced P4P effect likely to be due to within-teacher changes in skill or motivation. We find no evidence that recruits working under P4P made greater gains on the Grading Task from baseline to endline than did recruits working under FW. As already noted, recruits’ Dictator Game share sent is not a good predictor of teacher value added. But even if it were, we find no evidence that recruits working under P4P contributed more from baseline to endline than did recruits working under FW, if anything the reverse.

Instead, the experienced P4P effect is most plausibly driven by teacher effort. This conclusion follows from the arguments above and the direct evidence that recruits working under P4P provided greater inputs than did recruits working under FW. Specifically, the P4P contract encouraged recruits to be present in school more often and to use better pedagogy in the classroom, behaviors that were incentivized components of the 4P performance metric.

**Total effect** The total effect of the P4P contract combines both the advertised and experienced impacts: _τA_ + _τE_ . By the second year of the study, the within-year total effect of P4P is 0 _._ 04+0 _._ 16 = 0 _._ 20 standard deviations of pupil learning, which is statistically significant at the one percent level. Roughly four fifths of the total effect can thus be attributed to increased teacher effort, while the remainder arises from supply-side selection during recruitment. At a minimum, our results suggest that in relation to positive effort-margin effects, fears of pay-for-performance causing motivational crowd-out among new public-sector employees may be overstated.

Our estimates raise the question of why this effect is so much stronger in Year 2 compared to Year 1, particularly on the effort margin. One interpretation is that this is because it takes time for recruits to settle into the job and for the signal to noise ratio in our student learning measures to improve (Staiger and Rockoff, 2010). Consistent with this interpretation, we note that the impact of experienced P4P on incumbents did not increase in the second year. This interpretation suggests that 37
Year 2 effects are the best available estimates of longer-term impacts.

# **6 Conclusion**

This two-tier, two-year, randomized controlled trial featuring extensive data on teachers—their skills and motivations before starting work, multiple dimensions of their on-the-job performance, and whether they left their jobs—offers new insights into the compositional and effort margins of pay-for-performance. We found that potential applicants were aware of, and responded to, the first-tier labor market intervention. This supply-side response to advertised P4P was, if anything, beneficial for student learning. We also found a positive impact of experienced P4P that appears to stem from increased teacher effort, rather than selection-out or changes in measured skill or intrinsic motivation.

Given these encouraging results, it is natural to ask whether it would be feasible and cost effective to implement this P4P contract at scale. We worked closely with the government to design a contract that was contextually feasible and well-grounded in theory. A composite P4P metric was used to avoid narrowly emphasizing any single aspect of teacher performance and, when measuring learning, we followed the pay-for-percentile approach that aims to give all teachers a fair chance, regardless of the composition of the students they teach. We also took care to ensure that the P4P contract, if successful, could be built into the growth path of teacher wages. While a larger bonus might have elicited stronger impacts, the expected value of the P4P bonus was set at three percent of teacher salaries to be commensurate with annual teacher salary increments (and discretionary pay in other sectors under Rwanda’s _imihigo_ system of performance contracts for civil servants).

The fact that we compared a P4P contract with an expenditure-equivalent fixed wage alternative that is equal in magnitude to annual teacher salary increments means that it is reasonable to think about cost effectiveness primarily in terms of measurement. For pupil learning, the minimum requirement for the P4P contract we study is a system of repeated annual assessments across grades and key subjects.<sup>42</sup> Measurement of the other aspects of performance—teacher presence, preparation, and pedagogy—can in principle be conducted by head teachers or district staff (who are increasingly being asked to monitor teacher performance) at modest cost.

42 Such a system may soon be a part of Rwanda’s ‘comprehensive assessment’ program. See, e.g., https://www.newtimes.co.rw/opinions/mineducs-new-guide-student-assessment-triggers-debate.

38
There are nonetheless limitations of our work. Inasmuch as the impacts on either the compositional or effort margin might differ after five or ten years, there is certainly scope for further study of this topic in low- and middle-income countries. For instance, it would be interesting to explore whether long-term P4P commitments influence early career decisions to train as a teacher; our study restricts attention to employment choices by individuals who have already received TTC degrees.

Another set of issues relate to unintended consequences of pay-for-performance. We found that advertised P4P attracted teachers with lower intrinsic motivation, as measured by the share sent in the framed baseline Dictator Game. It is possible that the students taught by these more self-regarding teachers became more selfregarding themselves or otherwise developed different soft skills. We also found that experienced P4P improved performance on three of the four incentivized dimensions of the composite metric: teacher presence and pedagogy, and pupil learning. It is conceivable that the students in P4P schools may have been impacted by ‘multitasking’ as teachers focused on these dimensions to the detriment of others. Since we did not measure aspects of student development beyond test score gains, it would be interesting to explore these issues in future work.

Rwanda’s labor market has a characteristic that is unusual for low- and middleincome countries: it has no public sector pay premium, and consequently many of those qualified to teach choose not to, making it more similar to high- income country labor markets in this regard. Whether the positive effects we find in Rwanda of a multidimensional, pay-for-percentile contract—improving performance without dampening employee satisfaction—will generalize to settings where public-sector wage premiums differ remains an open question, for the education sector and beyond.

39