metadata: authors: 0: author: Barrera-Osorio,Felipe
1: author: Raju,Dhushyanth
count: Pakistan
docna: 0: docna: Teacher performance pay : Experimental
            evidence from Pakistan
docty: Policy Research Working Paper
subtopic: Administrative & Civil Service Reform,Public Sector Administrative & Civil
            Service Reform,Educational Sciences,Educational Institutions & Facilities,Inflation,Effective Schools and Teachers,Democratic Government,Public Sector Administrative and Civil
            Service Reform,De Facto Governments
teratopic: Education
abstracts: cdata!: This paper presents evidence from the
            first three years of a randomized controlled trial of a
            government-administered pilot teacher performance pay
            program in Punjab, Pakistan. The program offers yearly cash
            bonuses to teachers in a sample of public primary schools
            with the lowest mean student exam scores in the province.
            Bonuses are linked to three school-level indicators: the
            gain in student exam scores, the gain in school enrollment,
            and the level of student exam participation. Bonus receipt
            and size are also randomly assigned across schools according
            to whether or not the teacher is the school’s head. On
            average, the program increases school enrollment by 4.1
            percent and student exam participation rates by 3.4
            percentage points, both in the third year. The analysis does
            not find that the program increases student exam scores in
            any year. Mean impacts are similar across program variants.
            The positive mean impact on school enrollment is mainly seen
            in urban schools and the positive mean impact on student
            exam participation rates is only seen in rural schools.
geo_reg_key: 517198,517191,517212
repnme: repnme: Teacher performance pay : Experimental
            evidence from Pakistan
display_title: Teacher performance pay :
            Experimental evidence from Pakistan
listing_relative_url: /research/2015/06/24638080/teacher-performance-pay-experimental-evidence-pakistan
topicv3: Teachers Management m1326119 1523,Inflation m1326709 0870F4727ACED1CF8525815C00471347,Individual Performance Management m1326767 2058,Education m1327048 470,Teacher Performance m1327509 1531
publishtoextweb_dt: 2015-06-15T00:00:00Z
historic_topic: Education
no_of_pages: 37
author_keys: 0: author_key: 237905
model_extractions: WPS7307 Policy Research Working Paper 7307 Teacher Performance Pay Experimental Evidence from Pakistan _Felipe Barrera-Osorio Dhushyanth Raju_ South Asia Region Office of the Chief Economist June 2015
Policy Research Working Paper 7307

# **Abstract**

This paper presents evidence from the first three years of a randomized controlled trial of a government-administered pilot teacher performance pay program in Punjab, Pakistan. The program offers yearly cash bonuses to teachers in a sample of public primary schools with the lowest mean student exam scores in the province. Bonuses are linked to three school-level indicators: the gain in student exam scores, the gain in school enrollment, and the level of student exam participation. Bonus receipt and size are also randomly assigned across schools according to whether or not the teacher is the school’s head. On average, the program increases school enrollment by 4.1 percent and student exam participation rates by 3.4 percentage points, both in the third year. The analysis does not find that the program increases student exam scores in any year. Mean impacts are similar across program variants. The positive mean impact on school enrollment is mainly seen in urban schools and the positive mean impact on student exam participation rates is only seen in rural schools.

This paper is a product of the Office of the Chief Economist, South Asia Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at felipe_barrera-osorio@gse.harvard.edu and draju2@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Teacher performance pay: Experimental evidence from Pakistan

Felipe Barrera-Osorio Harvard Graduate School of Education Dhushyanth Raju World Bank _JEL classification codes_ : C93, I21, I28, M52 _Keywords_ : performance pay, incentive pay, public schools, teachers, field experiment, primary education, Punjab, Pakistan Barrera-Osorio: felipe_barrera-osorio@gse.harvard.edu; Raju: draju2@worldbank.org. We thank the Government of Punjab’s School Education Department and the Program Monitoring and Implementation Unit and, in particular, Iqbal Awan, Farah Masud, Naseem Nawaz, Sohail Raza, and Fiaz Ahmed Sajid for extensive collaboration with and assistance to the study team; Alejandro Ganimian for research assistance; and Robert Chase, Jishnu Das, Gladys Lopez-Acevedo, Richard Murnane, Quynh Nguyen, Laura Trucco, and Katja Vinha for comments and suggestions. We are also grateful for comments and suggestions by seminar participants at Colgate, Harvard, and Stanford Universities. The Government of Punjab’s Education Sector Reform Program, including the teacher performance pay program under study, received financial and technical assistance from the Canadian Department of Foreign Affairs, Trade, and Development (DFATD), the United Kingdom Department for International Development (DFID), and the World Bank. Financial support for the study from the Australian Department of Foreign Affairs and Trade (DFAT), the Weiss Family Program Fund for Research in Development Economics at Harvard University, and the World Bank is gratefully acknowledged.
# **I. Introduction**

Teacher effectiveness—a key determinant of school quality—is perceived to be poor in many low-income countries (Hanushek and Rivkin 2006, Glewwe and Kremer 2006). International development organizations increasingly promote performance (or incentive) pay to governments as an option to raise teacher effort and, thereby, teacher effectiveness.

Some recent teacher performance pay ventures in low-income countries have been rigorously evaluated. These evaluated interventions serve as valuable proof-of-concept demonstrations but are “special” in character. Although largely administered in public schools, the evaluated interventions were often designed and administered by interested and capable NGOs, with significant guidance and oversight from outside researchers. In addition, data for administering the interventions were gathered by the researchers and partnering organizations. (See, for example, Glewwe, Ilias, and Kremer 2010; Duflo, Hanna, and Ryan 2012; Muralidharan and Sundararaman 2011.) Hence, documented impacts may not be generalizable to other, different arrangements, such as where the government has overall primary responsibility for implementation (Bold, Kimenyi, Mwabu, Ng’ang’a, and Sandefur 2013).

We evaluate the _Improvers Bonus Program for Government School Teachers_ —a pilot public school teacher performance pay program conceived, designed, and managed by the provincial government of Punjab, Pakistan. The government is responsible for all aspects of the program and uses its own administrative data to assess teacher performance. While multiple international donor agencies provide financial and technical support to the government’s education reform agenda which includes the teacher performance pay program, the degree of external influence on the program does not differ from other, supported but unevaluated programs in the government’s reform agenda. Presumably, there is also less such influence than in the demonstrations we noted above.

Our study contributes evidence on the effectiveness of teacher performance pay programs, which fall under the class of incentive-based, supply-side education interventions.<sup>1</sup>

> 1 Available rigorous evidence on the impacts of teacher performance pay is inconclusive: in some studies, there is consistent evidence of positive impacts (see, for example, Lavy 2002; Duflo, Hanna, and Ryan 2012; Muralidharan and Sundararaman 2011; and Muralidharan 2012); in others, the evidence is mixed or conditional (see, for example, Glewwe et al 2010; Behrman, Parker, Todd, and Wolpin 2012; Sojourner, Mykerezi, and West 2014; and Fryer, Levitt, List, and Sadoff 2012; and yet in others, there is no evidence of positive impacts (see, for example, Fryer 2013; Springer et al 2011; and Goodman and Turner 2013).

2
More generally, our study contributes evidence on the effectiveness of public service delivery reform in a low-income setting where government capacity and accountability are considered to be poor.

Initiated in mid 2010, the program offers yearly cash bonuses linked to school performance to public primary school teachers. At program registration, teachers are offered some basic school management, teaching, and exam preparation tips. The program is however designed to incentivize incumbent teachers to raise school performance by increasing their effort rather than by directly increasing their skill.<sup>2</sup> The bonuses are offered to teachers on top of their standard salaries, and are set as a linear function of a composite score of school performance. The composite score is obtained from a weighted sum of three indicators:

- the gain in the school’s enrollment in first through fifth grade,

- the gain in the school’s mean score on Punjab’s standardized fifth-grade examination, and

- the participation rate of the school’s fifth-grade students in the examination.

The first two indicators are consistent with the government’s main education goals: to raise school participation and student academic achievement. Participation and achievement are both acutely low in Punjab. The exam participation rate is included as a deterrent against potential exampool selection by program schools. The data for the indicators come from the Punjab Examination Commission (PEC) fifth-grade exam, which is a yearly standardized exam, and the Annual School Census (ASC) survey, which is a yearly field survey of all public schools. Both data instruments are administered by the government.

Increasing the effort of head teachers may be an important way to raise school performance, and bonuses to head teachers may help elicit greater effort from them. Studies find

> In spite of these mixed results, a few general patterns can be drawn from the collective evidence. Programs tend to have targeted schools with low baseline levels of student academic achievement. Program impacts are usually modest if present, but are large in a few cases. Programs in low-income countries have a higher success rate than those in high-income countries. Discussions of program implementation, in particular the fidelity and quality of implementation, are limited, if any. There is little or no rigorous evidence on, where relevant, the conditions and pathways behind intended (and unintended) impacts or the factors behind the lack of intended impacts.

> 2 Growing credible evidence suggests the prevalence and severity of low teacher effort, of which two basic measures are whether the teacher is present in school and whether the teacher is on-task. Teacher absentee rates are found to be high in many low- and middle-income countries, and appear to result from many teachers being occasionally absent rather than a few teachers being frequently absent. When teachers are present in school, a large share of them are found to be off task (Glewwe et al 2010; Chaudhury, Hammer, Kremer, Muralidharan, and Rogers 2006).

3
that the performance of head teachers—as school leaders—is an important determinant of school performance (see, for example, Waters, Marzano, and McNulty 2003; Robinson, Claire, Lloyd, and Rowe 2008; Mulford 2003).<sup>3</sup> Bonuses to head teachers may also be a more efficient way to raise school performance than providing bonuses to all teachers. To test whether leveraging head teachers is either sufficient or augments program impacts, the receipt and size of the bonus for a given level of school performance are randomly assigned across schools according to whether or not the teacher is the school’s head teacher.

In the first treatment variant, head teachers are eligible for “level-1” bonuses, while other teachers are not eligible for any bonuses. In the second treatment variant, all teachers (whether head teacher or not) are eligible for level-1 bonuses. In the third and last treatment variant, head teachers are eligible for higher “level-2” bonuses, while other teachers are eligible for level-1 bonuses. Level-2 bonuses are fixed to be twice as large as level-1 bonuses for a given level of school performance. The treatment variants and untreated status are randomly assigned across the evaluation sample of 600 public primary schools with essentially the lowest school-level mean 2010 PEC exam scores, located in three districts (out of Punjab’s 36 districts) with the lowest district-level mean 2010 PEC exam scores.

Using the PEC exam and ASC survey data, we evaluate program impacts on school enrollment, exam participation rates, and student exam scores in the first three years of program implementation. Evidence of significant mean impacts across the three indicators is mixed. Significant positive mean impacts are concentrated in the third year. The program increases school enrollment by, on average, 4.1 percent in the third year. Treatment variants have similarly-sized third-year mean impacts, but only the impact for the treatment variant that offers bonuses solely to head teachers is significant. The program increases exam participation rates by, on average, 3.4 percentage points (ppts) in the third year. Given a baseline mean exam participation rate of 97 percent, the mean impact maximizes the rate. Treatment variants have similarly-sized third-year mean impacts, and all of them are significant. We do not find that any of the treatment variants, and hence the program, increases student exam scores in any year.

> 3 Factors that may drive a positive relationship between head teachers and school performance include “instructional leadership” (the ability to create an environment conducive for student learning), “transformational leadership” (the ability to motivate and empower teachers), and “managerial skills” (the ability to manage the school as a typical bureaucracy) (Boyd at al 2011, Ingersoll 2011, Grissom and Loeb 2011). High-performing school systems are often found to nurture, retain, and promote leaders in their teaching workforces as a core strategy for student learning (see, for example, Darling-Hammond and Rothman 2011 and World Bank 2011).

4
We test for two forms of strategic manipulation that could explain the mean impacts on school enrollment and exam participation rates. First, program schools may choose to reduce fifth-grade enrollment by limiting promotion to fifth grade or shedding fifth-grade students in order to raise their exam participation rates. We do not find consistent evidence across treatment variants that the program decreases fifth-grade enrollment. Second, program schools may choose to inflate reported enrollment. By constructing pseudo-panel cohorts using grade-specific enrollment information from the yearly ASC surveys, we test whether the program flattens the gradient at which enrollment declines with grade (or, more extremely, inverts it), which would be consistent with enrollment inflation. We do not find evidence suggesting this.

Post-hoc subgroup analysis reveals that the third-year program mean impacts on school enrollment and exam participation rates vary between urban and rural schools. The program mean impact on enrollment in the total sample is mainly due to the urban subsample (which represents 8 percent of the total sample). The program increases enrollment in urban schools in the third year by, on average, 23 percent (the third-year mean impact in rural schools is 2 percent and insignificant). Looking at enrollment separately by grade, we find significant positive mean impacts of similar size in percent terms across first through fourth grade, indicating higher intake as well as higher retention. The program mean impact on exam participation rates in the total sample is fully due to the rural subsample. The program increases exam participation rates in rural schools in the third year by, on average, 3.6 ppts (the third-year mean impact in urban schools is –1.4 ppts and insignificant). Program mean impacts on student exam scores differ between the urban and rural subsamples in the first year but the differences in later years are not significant.

<mark>Given that the program increases exam participation rates, we test whether the absence of significant positive mean impacts on student exam scores is due to negative selection of students into examtaking. We do not find evidence suggesting this. Apart from this test, we are left to speculate on factors that could have blocked impacts.</mark> We conclude the paper with this speculative discussion but, to summarize, <mark>we d</mark> iscount that the absence of positive mean impacts is due to program implementation failures. We also discount that it is due to bonus awards being too low, to the bonus formula being too complex for teachers to comprehend, or (as relevant for the treatment variant) to the group-based nature of the bonuses inducing significant freeriding.

5
We posit three hypotheses for the absence of positive mean impacts on student exam scores. First, nonschool or school factors outside the control of teachers may make raising student exam scores difficult even with greater teacher effort. Targeting schools with the lowest mean student exam scores in the province places the program in circumstances that may be especially handicapping. Second, teachers may lack the knowhow to raise student exam scores. Third, teachers may optimally choose to direct their effort at those incentivized margins that maximize payoffs net of effort costs. In this case, it appears that the net payoff-maximizing margin is school enrollment in urban areas, exam participation rates in rural areas, and not student exam scores anywhere.

The rest of the paper is organized as follows. Section II describes the context and the program. Section III discusses the data, sample, and empirical strategy. Section IV reports the empirical results. In concluding, Section V summarizes the main results and discusses potential explanations for the absence of program impacts on student exam scores.

# **II. Context and program**

# _A. Context_

Punjab, the site of the teacher performance pay program, is the largest of Pakistan’s four provinces, accounting for three-fifths of the country’s population and income. In 2012/13, the estimated net enrollment rate at the primary school level was 66 percent (2012–13 Pakistan Social and Living Standards Measurement [PSLM] survey report, Pakistan Bureau of Statistics, Government of Pakistan). In 2007, third-grade students in a school sample in rural Punjab scored, on average, 31 percent in English, 27 percent in Urdu, and 34 percent in mathematics on independently administered, competency-based tests. These results are significantly below official grade-level standards.<sup>4</sup> The public school system is the main provider of education in Punjab. In 2012–13, 61 percent of primary school students were in public schools (2012–13 PSLM report, Pakistan Bureau of Statistics, Government of Pakistan). The system is large, composed of 51,504 functional schools. Seventy percent of these schools are primary schools (preschool to fifth grade) with 105,300 assigned teachers and 2.7 million children enrolled in first through fifth

> 4 Statistics provided by Jishnu Das.

6
grade (October 2013 Annual School Census, School Education Department, Government of Punjab).

Available evidence indicates that teacher performance in public schools is poor. Using field survey data from rural Punjab, Andrabi, Das, Khwaja, Vishwanath, and Zajonc (2009) cast light on the performance of public school teachers, particularly when contrasted with teachers in low-cost, for-profit private schools operating in the same villages. They find that public school teachers tend to have higher levels of academic qualifications, teaching experience, and professional training than do private school teachers. Public school teacher salaries are, on average, three to four times higher than private school teacher salaries after accounting for differences in teacher credentials and other characteristics.

Despite the higher levels of credentials among public school teachers, Andrabi et al find that the levels of teacher presence and student academic achievement are lower in public schools than in private schools. It also appears that public school teacher salaries do not vary with teacher competency (measured by teacher test scores) or student competency (measured by student test scores). Higher public school teacher salaries also seem to be associated with lower teacher effort, as measured by teacher presence at school. This pattern is driven mainly by the relatively lower presence of more senior (and, therefore, better paid) teachers in public schools. In contrast, among private school teachers, higher salaries are associated with higher teacher presence, higher teacher test scores, and higher student test scores.

# _B. Program_

The teacher performance pay program was designed and approved by the Punjab government in fiscal year (FY) 2009–10 for implementation on a pilot basis starting in FY2010– 11 (corresponding to the 2010–11 school year).<sup>5</sup> The main expressed aim of the program is to raise school performance by encouraging greater teacher effort.

# _Measuring school performance_

The government selected performance indicators which could be measured regularly, throughout the province, in a standardized way across schools, and for which the underlying data

> 5 The government’s fiscal year begins on July 1 and ends on June 30.  The public school system’s school year begins on April 1 and ends on March 15.

7
were considered least susceptible to manipulation by schools. The government possessed two data sources that fulfilled these criteria: the Annual School Census (ASC) surveys and the Punjab Examination Commission (PEC) exams.

Initiated in 2003, the ASC is an annual field survey of public schools which captures basic school and teacher information, including student enrollment by grade and gender, with a reference date of October 31. Initiated in 2006, the PEC exam is an annual, standardized academic test which is mandatory for public school students in fifth and eighth grades. The test uses selected and constructed response questions based on standard learning objectives of Pakistan’s official curriculum. Core subjects are English, Urdu, Islamic studies, mathematics, science, and social studies, all assessed through pencil-and-paper tests.

The provincial education department is responsible for managing all stages of the ASC survey and PEC exam activities, and is solely responsible for the design of the data collection instruments. At the field level, Monitoring and Evaluation Assistants (MEAs) visit assigned public schools in November to collect ASC survey data. The MEAs are mostly retired members of the armed services and report directly to the provincial education department. The fifth-grade PEC exam is administered in January and/or February by district education department staff and enlisted public middle and high school teachers. Public middle and high schools serve as exam centers. Filled-in multiple choice scoring sheets are dispatched to PEC headquarters to be read by optical mark readers. Answers to open-ended questions are scored by recruited public middle and high school teachers at grading centers in the district; the scores are transmitted to PEC headquarters.

The provincial education department selected three indicators for measuring school performance using the ASC survey and PEC exam data:

- the gain in mean exam scores ( _GTS_ ),

- the gain in enrollment in first through fifth grade ( _GSE_ ), and

- the exam participation rate ( _TPR_ ).

_GTS_ is defined as the year-over-year gain in the school’s percent mean score in the fifth-grade PEC exam. The indicator is measured in percentage point terms. The exam scores used by the government for the indicator were raw scores—the number of exam questions answered 8
correctly—for the multiple choice question modules in the core subjects.<sup>6</sup> _GSE_ is defined as the year-over-year gain in the school’s enrollment in first through fifth grade captured in the ASC survey. The indicator is expressed in percent terms. _TPR_ is defined as the number of students who took the fifth-grade PEC exam from a school (recorded by PEC) divided by the number of enrolled students in fifth grade in the school (captured in the ASC survey). The indicator is expressed in percent terms. The values for each of the three indicators are bounded by zero percent and 100 percent.

A composite school performance score ( _CS_ ) was formulated as follows: _CS_  0.65 _GTS_  0.25 _GSE_  0.15 _TPR_ .(1) The assigned weights reflect the provincial government’s relative valuation of the three indicators in assessing school performance.

# _Targeting_

The program is targeted at public primary schools with among the lowest mean student exam scores in the province following a three-step process. First, the three districts with the lowest district-level mean scores in the 2010 fifth-grade PEC exam in Punjab’s 36 districts were selected. These districts were Attock, Mandi Bahuaddin, and Rahimyar Khan. Second, within these districts, schools at the primary level that were functional according to the October 2009 ASC survey and had at least ten students who took the 2010 fifth-grade PEC exam were identified. Third, 1,962 such schools were ranked from highest to lowest by their 2010 schoollevel mean fifth-grade PEC exam scores; the lowest ranked 600 schools were selected for the evaluation sample. The range of mean scores for the sample corresponds to that of 12 percent of all public primary schools in the province.

# _Treatment variants_

In principle, the teacher assigned to be (acting) head teacher is authorized by the district education department to manage the school and its teachers. Head teachers can manage teachers by supervising, rewarding, sanctioning, and supporting teachers. If head teachers are a

> 6 Raw scores are used instead of the scores scaled by PEC because of concerns related to the conversion formula that is applied. The scoring of answers to multiple-choice questions is considered more reliable than the scoring of answers to open-ended questions.

9
sufficiently effective instrument for managing teachers and their performance, offering bonuses to head teachers may be a more efficient way to raise school performance than offering bonuses to all teachers in the school.

These views shaped the design of three alternative treatment variants to test in parallel. Bonus eligibility and size were varied according to whether or not the teacher is the school’s head teacher.<sup>7</sup> The treatment variants are:

- **_HT only_** : only head teachers were eligible for level-1 bonuses.

- **_all T_** : both head teachers and teachers were eligible for level-1 bonuses.

- **_HT+_** : head teachers were eligible for level-2 bonuses, while other teachers were eligible for level-1 bonuses.

The size of the level-2 bonus was set as twice the size of the level-1 bonus for a given level of school performance. Treatment variants and untreated status are randomly assigned across the evaluation sample by first blocking schools into quads based on 2010 school-level mean fifthgrade PEC exam scores.<sup>8</sup>

# _Bonus formula_

Table 1 presents the bonus formula, by treatment variant and teacher type (head teacher or other teacher). The slope parameters in the bonus formula were calibrated so that teachers in the average school at baseline in the evaluation sample face high-powered incentives to perform. Even if the average school does not increase enrollment and mean student exam scores but maintains its baseline exam participation rate of 97 percent, the minimum teacher bonus payout would be 14,550 Pakistani rupees (US$169).<sup>9</sup> If the average school maxes out the composite score in the first year, the minimum teacher bonus payout would be 86,200 Pakistani rupees (US$1,002).<sup>10</sup> These two minimum payouts represent 10 and 57 percent respectively of the

> 7 A teacher appointed as a head teacher on an acting basis is considered eligible for the head teacher bonus. A (head) teacher whose assignment to a program school is effective at the start of the school year is considered eligible for the (head) teacher bonus even if the (head) teacher is then transferred at any point during the school year to any another school.

> 8 The trial is unmasked: the experimental groups that schools are assigned to are known by the provincial and district education departments, program implementing partners, supporting international donor agencies, and us (the researchers).  Program schools are directly made aware of their assigned treatment variant.

> 9 We use the December 2010 exchange rate of 86 Pakistani rupees per US dollar.

> 10 Maxing out the composite score in the first year would imply raising the exam participation rate from 97 to 100 percent, doubling enrollment from 100 students, and raising the school mean exam score from 23 to 100 percent.

10
yearly basic salary of an average teacher at baseline in the evaluation sample.<sup>11</sup> The percentages would be two-thirds to one-half their size if we instead use take-home pay (basic pay plus cash benefits) as the base.<sup>12</sup>

# _Implementation_

In each of the three program years (FY2010–11, FY2011–12, and FY2012–13), staff from the provincial education department and partnering organizations conducted a field visit in the middle of the school year. During this visit, program school teachers were invited to attend meetings, separately by treatment variant-specific, proximity-based school groupings.

The main aim of the meetings was to acquaint teachers with their relevant treatment variant and register them for the program. To register, the teacher filled in a form providing employment details and bank account information. The meetings were also used to facilitate experience sharing between teachers and to offer some tips for basic school and class management, teaching, and exam preparation.

At the end of the first and second program years (two months after the end of the school year), school performance score cards were sent from the provincial education department to the district education departments. The score cards reported the school’s _GTS_ , _GSE_ , _TPR_ , and _CS_ values for the school year that just ended. The district departments were instructed to dispatch the cards to program schools and to have them displayed in a prominent, open location on school premises. These cards were printed on large, durable (synthetic flex) paper.

At the end of the third program year, the provincial education department and partnering organizations held meetings with program school head teachers and directly handed them score cards (with a new format) to take back and display at their schools. This round of meetings was also used to facilitate experience sharing between teachers and to reiterate offered tips.

In each program year, school performance scores and bonus awards were calculated by the provincial education department, with technical support and quality control by partnering organizations. The bonus awards, aggregated up to the district level, were communicated to the

> 11 The average teacher is at pay grade 9 and has 17 years of government service. The 2011 pay schedule for the Government of Pakistan indicates that an official at this pay grade and with this service length would earn a yearly basic salary of 151,920 Pakistan rupees (US$1,767).

> 12 Government school teachers receive multiple monthly cash benefits (or “allowances”) for, among other things, transportation, medical, and housing expenses, and cost-of-living relief.

11
provincial finance department. The provincial finance department transferred the requested total bonus award for a district to an account dedicated to the program operated by the district education department. At the same time, the provincial education department sent the district education department a list of program schools and teachers with the bonus award and the personal bank account information of program school teachers.

Provincial government and third-party validations find that program implementation has been generally satisfactory since the first year of implementation.

# _Comparison with advocated best practice_

The program’s design takes account of key considerations and issues discussed in the literatures on optimal incentive contracting and incentive pay (see, for example, Neal 2011 for a review of performance pay in education) in at least four ways. First, the program’s piece-rate bonus structure arguably incentivizes efficient effort from teachers who may differ in their level of skill. Second, the program links bonuses to the exam participation rate to protect against teachers’ selecting academically-stronger students to register and take the exam. Indeed, highstakes accountability and performance pay programs are often documented to result in gaming (see, for example, Figlio and Winicki 2005 and Jacob and Levitt 2003). Third, in two of the three treatment variants, the program offers school-level group bonuses which can help prevent perceptions of unfairness and unproductive competition between teachers in the same school, although group bonuses can create a freerider problem (Holmstrom 1982; Cohn, Fehr, Herrmann, and Schneider 2013). Fourth, the program attempts to avoid a potential multitasking problem (Baker 1992, Holmstrom and Milgrom 1991). The program links bonuses to changes in school-level mean student exam scores to incentivize program school teachers to exert effort across all students and perhaps greater effort on relatively weaker students with more room to improve as well as incentivizes all program schools (irrespective of their baseline mean exam scores) to raise their performance (Barlevy and Neal 2012).<sup>13</sup> The program also links bonuses to student exam scores from the five core subjects in the country’s curriculum to avoid teachers’ shifting their effort to a narrow set of incentivized subjects (Jacob 2005, Koretz 2002).

> 13 This structure contrasts with one that links bonuses to a minimum test score which can distort teacher effort towards students near the minimum test score or links bonuses to a minimum test pass rate which can discourage teacher effort in schools that are relatively farther away from the minimum test pass rate.

12
Nevertheless, the program’s design is imperfect in at least three ways. These imperfections, either singly or jointly, are not unique to this program. First, the program does not use teacher value-added measures which adjust gains in student exam scores for existing differences in relevant student and school characteristics (Koedel, Mihaly, and Rockoff _forthcoming_ ).<sup>14</sup> Second, while the ASC survey and PEC exam data are used for both monitoring and accountability, the program makes these data more high stakes, increasing the risk that gaming and data manipulation undermine data reliability (even if this risk is limited to the relatively small sample of program schools). Third, gains in student exam scores may be due to teachers coaching students in examtaking strategies rather than teachers engaging in actions that produce actual student learning.<sup>15</sup>

# **III. Data, sample, and empirical strategy**

# _A. Data and sample_

Four annual rounds of the ASC survey and PEC exam data were linked together using the government’s unique school identification codes (the Education Management Information System [EMIS] code). The October 2009 ASC survey and the January–February 2010 PEC exam rounds serve as our baseline data. (The program came into effect in July 2010.) The October 2012 ASC survey and January–February 2013 PEC exam rounds serve as our data for the third program year.

Evaluation sample schools were linked largely without issue across the various databases. We could not link a few schools because they were merged into other schools as part of Punjab’s school merger program.<sup>16</sup> As a result, 583 schools (or 97 percent of the original evaluation sample) remain in the evaluation sample in the third program year. The shares of schools “attrited out” from the original evaluation sample do not differ between treated and untreated samples (2.7 percent versus 3.3 percent, respectively).

> 14 The use of value-added measures for teacher performance management is however contentious in education research and practice communities (see, for example, Goldhaber 2010).

> 15 Neal (2011) argues that exam preparation can be defensible if performance pay shifts teachers from ineffective or lack of teaching—which presumably applies in our case (see Chaudhury et al 2006 for evidence from low- and middleincome countries on teacher absentee and off-task rates)—to exam preparation than from effective teaching to exam preparation.

> 16 Acquired schools lose their status as separate schools and are assigned the same EMIS codes as their respective acquiring schools.

13
Column 1 in Table 2 reports baseline mean outcomes and characteristics for schools in the original evaluation sample. Eight percent of schools are urban. Eighty-six percent of schools have buildings in satisfactory condition. On average, schools have 3.1 out of 4 basic amenities (drinking water, electrical connection, toilets, and sewer connection). On average, schools have 3 teachers, with an average of 17 years of experience in government service and 10 years of service in their currently-assigned schools. Fifty-one percent of teachers have at most a high school diploma, 35 percent have a bachelor’s degree, and 14 percent have at least a master’s degree. On average, schools have 100 enrolled students in first through fifth grade, studentteacher and student-classroom ratios higher than 50:1, and an exam participation rate of 97 percent. School mean exam scores are standardized using the mean and standard deviation (sd) for school mean exam scores for untreated schools.

We compare schools in the original evaluation sample to all public primary schools in the province. Column 2 in Table 2 reports baseline mean outcomes and characteristics for 42,438 functional primary schools in the province (for ease, referred to as “all schools”) and Column 3 reports differences in baseline means between evaluation sample schools and all schools.

On average, evaluation sample schools were established earlier (+3 years). Evaluation sample schools are more likely to have buildings in satisfactory condition (+3 ppts); they have, on average, more classrooms (+0.4) and basic amenities (+0.3). Evaluation sample schools are just as likely to be urban as all schools, and less likely to be de-facto girls only (–15 ppts) and more likely to be de-facto coeducational or boys only. Teachers in evaluation sample schools are more likely to hold a bachelor’s degree than teachers in all schools (+11 ppts), and less likely to have completed high school (–9 ppts) or hold at least a master’s degree (–3 ppts).

Evaluation sample schools are, on average, larger (+22 students in first through fifth grade, +7 students in fifth grade), and more crowded (+8.3 students per classroom, +11.9 students per teacher). Finally, evaluation sample schools have, on average, higher exam participation rates (+4.4 ppts) and much lower exam scores (–4.2 sds). The last finding results from the way the evaluation sample is set.

# _B. Empirical strategy_

The evaluation is based on a randomized block design. The 600 evaluation sample schools were randomly assigned into the four treatment statuses ( _HT only_ , _all T_ , and _HT+_ , and 14
untreated) of 150 schools each, after blocking by baseline school mean exam scores. This design ensures that any differences in mean outcomes between treatment statuses are attributable to differences in treatment statuses and not due to differences in baseline mean outcomes and characteristics between treatment statuses.

We obtain mean impacts (specifically, intent-to-treat effects) by estimating via ordinary least squares an outcome regression of the form where<sup>_Y_</sup> _s_<sup>denotes the outcome in school</sup><sup>_s_,</sup> _Ti_ , _s_<sup>,</sup> _i_  1,2,3, treatment variant indicators (corresponding to treatment variants _HT only, all T_ and _HT+_ ), _X s_ ,0 baseline characteristics, and  _d_<sup>district indicators. We estimate alternative variants of (2). We estimate regressions where we</sup> pool the three treatment variants (<sup>_T_</sup> _s_<sup>equals one if the school is treated, and zero if untreated), as</sup> well as regressions where each treatment variant is included additively. We estimate separate regressions for each program year. Regressions for enrollment and exam participation rates are at the school level and estimated standard errors are clustered at the tehsil (sub-district) level; regressions for exam scores are at the student level and estimated standard errors are clustered at the school level.<sup>17</sup> Table 2 reports baseline mean outcomes and characteristics for the original evaluation sample, separately by pooled treatment status (treated schools in Column 4 and untreated schools in Column 5), and differences in means between treated and untreated schools (Column 6). With the exception of the de-facto gender type of the school and years of government service of the teacher, results of tests of pairwise differences in baseline means between treated and untreated schools are insignificant. Results of tests of joint differences in baseline means between treated and untreated schools for three groups of variables, namely school-, teacher-, and enrollment and exam score-related (Panels A, B, and C), are also insignificant.<sup>18</sup> In sum, the results confirm that treated and untreated school samples are balanced at baseline.

> 17 The country has four tiers of government administration: province, district, tehsil, and union council. The lowest level of public school system administration is the tehsil.

> 18 We also check balance between treated and untreated schools in baseline mean outcomes and characteristics for schools that remain in the evaluation sample in the third program year. This test is important because, as we will show,

15
# **IV. Results**

# _A. Program impacts_

Table 3 reports estimated mean impacts on school enrollment (Columns 1–3), exam participation rates (Columns 4–6), and student exam scores (Columns 7–9) in each of the three program years.

First, with respect to school enrollment, mean impacts of the treatment variants either turn from negative to positive or are positive and grow in size over the three years. Third-year mean impacts are 3.9 students for _HT only_ , 2.5 students for _all T_ , and 5.8 students for _HT+_ , and are not statistically different from each other. However, only the mean impact for _HT only_ is significant. The pooled treatment has a significant mean impact of 4.1 students (baseline mean enrollment is 100 students).

Second, with respect to exam participation rates, mean impacts of the treatment variants also either turn from negative to positive or are positive and grow in size over the three years. Third-year mean impacts are 3.4 ppts for _HT only_ , 2.8 ppts for _all T_ , and 3.6 ppts for _HT+_ , and are not statistically different from each other. The mean impacts for all treatment variants are significant. The pooled treatment has a significant mean impact of 3.3 ppts. Given a baseline exam participation rate of 97 percent, the mean impact maximizes the rate.

Third, with respect to student exam scores, mean impacts of the treatment variants do not exhibit a monotonic pattern of change over the three years. Only treatment variant _HT+_ has consistent positive mean impacts of .09, .14, and .05 sds in the first, second, and third years, but they are all insignificant. Mean impacts across the years for the other two treatment variants are also all insignificant. The pooled treatment has mean impacts of .02, .08, and –.02 sds in the first, second, and third years, but they are all insignificant.

# _Potential strategic manipulation of enrollment data_

The significant positive mean impacts on school enrollment in the third year may be due to program schools inflating enrollment. While schools do not have direct control over ASC

> we find significant mean impacts mainly in the third year. Results from pairwise and joint tests suggest that baseline means do not differ between treated and untreated schools in the attrited sample.

16
survey data, MEAs capture enrollment figures from school enrollment registers, which are under the control of schools.<sup>19</sup> Children mainly join public primary schools in preschool and first grade; re-entrants or transfers into higher grades are rare and dropouts much more frequent. Hence, we expect that enrollment in grade _g_ in year _y_ is equal to or less than enrollment in grade _g_  1 in year _y_  1. Enrollment inflation in year _y_ can flatten or invert this gradient. We test for this by estimating an OLS regression of the form where _Eg_ ,3, _s_ denotes enrollment in grade _g_ in the third program year in school _s ,_<sup>_T_</sup> _s_<sup>the pooled</sup> treatment indicator, _Eg_  _t_ ,3  _t_ ,s<sup>enrollment in grade</sup><sup>_g_</sup>  _t_ in program year 3  _t_ in school _s_ , where _t_ denotes a one-year  _t_  1  or two-year split  _t_  2  , 1  _t_  2  an indicator for a two-year time split,  _g_ grade _g_ indicators, and<sup></sup> _s_<sup>school indicators. A positive</sup> 1<sup>is taken to indicate inflation</sup> in untreated schools and a positive<sup></sup> 2<sup>the differential inflation in treated schools. We find</sup> <sup>ˆ</sup> 1 .53  _p_  .000  and <sup>ˆ</sup> 2 .03  _p_  .703  . Thus, we do not find evidence consistent with potential enrollment inflation.

_Potential strategic manipulation of exam grade size_ Increased exam participation rates may be due to more fifth-grade students taking the exam (an increase in the fraction’s numerator) or to fewer fifth-grade students (a decrease in the fraction’s denominator), or to both. Exam grade size can decline if teachers hold back students in earlier grades or if teachers expel fifth-grade students (although the bonus’s link to school enrollment would deter such behavior).<sup>20</sup> Evidence of a negative mean impact on fifth-grade enrollment would be consistent with strategic manipulation of exam grade size by program school teachers.

> 19 The provincial education department reports that MEAs crosscheck school enrollment figures against the actual presence of students but specific written guidelines are not provided.

> 20 Holding back or expelling academically-weaker students can provide the added benefit of higher student exam scores.

17
Table 4 reports estimated mean impacts on fifth-grade enrollment in each of the program years. Mean impacts are negative for each of the treatment variants and the pooled treatment in the third year. However, only the mean impact for _HT only_ of –1.1 students (or –6 percent given a baseline mean fifth-grade enrollment of 18.3 students) is significant. The pooled treatment’s mean impact of –.8 students is insignificant. Thus, we find inconsistent evidence of strategic manipulation of exam grade size by program schools.

# _Potential negative selection into examtaking_

We find that the program increases exam participation rates in the third year. If the program induces weaker students to participate in the exam, the mean impact on student exam scores will be biased downwards.

Adopting the approach by Angrist, Bettinger, and Kremer (2006), we re-estimate mean impacts on student exam scores by fitting a Tobit model to the data, artificially censoring the student exam score distribution at selected alternative low percentiles. All exam takers with scores below the censoring percentile are assigned the score at the censoring percentile.

Assuming that (the few) exam nontakers would have scores below the artificial censoring point if they had taken the exam, exam nontakers are also assigned the score at the censoring percentile. Recall that mean impacts on student exam scores estimated via ordinary least squares (reported in Table 3) are insignificant. If mean impacts estimated via Tobit are positive and significant, it suggests that negative selection into examtaking may be behind the least squares results.

Table 5 reports third-year mean impacts on student exam scores estimated via Tobit at four different censoring percentiles: 5<sup>th</sup> , 10<sup>th</sup> , 15<sup>th</sup> , and 20<sup>th</sup> . Mean impacts for the treatment variants, and the pooled treatment, are largely stable across the four censoring percentiles and are all insignificant. Thus, we do not find evidence of potential negative selection into examtaking.

# _B. Program impacts by location_

In post-hoc subgroup analysis, we examine whether mean impacts vary by school location.<sup>21</sup> Location is defined as whether the school is urban or rural. Only 48 schools (or 8

> 21 We also examine whether mean impacts vary by baseline mean exam scores, and find that they generally do not. Results are available upon request.

18
percent) are in the urban subsample, and, hence, the analysis of their mean impacts may be underpowered.

Table 6 reports baseline mean outcomes and characteristics for urban and rural schools (Columns 1 and 2), and pairwise differences in means between the two subsamples (Column 3). Compared to rural schools, urban schools have a higher mean number of school amenities (+.3 out of 4 amenities), mean number of teachers (+2.3 teachers), mean number of students in first through fifth grade (+40.2 students), mean number of students in fifth grade (+6.7 students), mean years of government service for teachers (+3.4 years), and mean exam scores (+.4 sds). Urban schools also have a lower mean student-teacher ratio (–18 students per teacher) than rural schools.

Table 6 also reports differences in baseline mean outcomes and characteristics between treated and untreated schools in the urban and rural subsamples (Columns 4 and 5). Treated and untreated schools differ in two dimensions in the urban subsample and one dimension in the rural subsample. The differences in means, when present, are large: the share of schools with buildings in satisfactory condition is 18.4 ppts lower in treated than in untreated schools in the urban subsample; the share of coeducational schools is 11.2 ppts lower in treated than in untreated schools in the rural subsample. Nevertheless, we cannot reject that the differences in means are jointly equal to zero for our school-, teacher-, and enrollment and exam score-related groupings of dimensions (Panels A, B, and C) in the urban and rural subsamples.

We test for differential mean impacts between urban and rural schools by estimating an OLS regression of the form where _U s_ denotes whether or not the school is urban, and all other variables are as defined previously in (2). The mean impact for rural schools is given by<sup></sup> 1<sup>, the differential mean impact</sup> for urban schools by  3 , and the net mean impact for urban schools by  1   3 .

As before, we estimate alternative variants of (3). We estimate regressions (a) where the three treatment variants are pooled, (b) where the treatment variants are included additively, and (c) separately for each of the three program years. Regressions for enrollment and exam participation rates are at the school level and standard errors are clustered at the tehsil level; 19
regressions for exam scores are at the student level and standard errors are clustered at the school level.

<mark>Table 7 reports estimated differential mean impacts by school location.</mark> Mirroring the findings for the overall evaluation sample, significant mean impacts are concentrated in the third year. First, with respect to school enrollment, third-year mean impacts for the urban subsample relative to the rural subsample are an additional 34 students for _HT only_ , 27 students for _all T_ , and 29 students for _HT+_ , but are significant only for _HT only_ and _all T_ . The pooled treatment has a significant third-year mean impact on enrollment of an additional 30 students in the urban subsample. The pooled treatment’s _net_ third-year mean impact in the urban subsample is 32 students (or 23 percent given a baseline mean enrollment of 137 students in urban schools).

Second, with respect to exam participation rates, third-year mean impacts in the rural subsample are 3.7 ppts for _HT only_ , 3.0 ppts for _all T_ , and 4.2 ppts for _HT+_ , and are all significant. The third-year net mean impacts of the treatment variants in the urban subsample are all negative and insignificant. The pooled treatment has a significant third-year mean impact of 3.6 ppts in the rural subsample (and an insignificant third-year net mean impact of –1.4 ppts in the urban subsample).

Third, with respect to student exam scores, third-year (differential) mean impacts of the treatment variants and the pooled treatment are insignificant for the rural (urban) subsamples. The pooled treatment however has a significant first-year differential mean impact of .34 sds in the urban subsample. While the corresponding mean impacts are .37 sds for _HT only_ , .29 sds for _all T_ , and .34 sds for _HT+_ , only the impact for _HT only_ is significant.

We do not find that the differential mean impact on school enrollment for the urban subsample is due to enrollment inflation. Nor does it appear that the mean impact on exam participation rates for the rural subsample is due to manipulation of exam grade size. We check third-year means of basic school and teacher characteristics for treated and untreated schools separately for the urban and rural subsamples. In each subsample, the means are largely similar, thus failing to suggest potential pathways behind detected impacts.<sup>22</sup> Lastly, we examine how the third-year mean impact on school enrollment in the urban subsample is spread across grades. Table 8 reports estimated differential mean impacts of the

> 22 Results for the two tests of potential strategic behavior and the estimated third-year means of basic school and teacher characteristics are available upon request.

20
pooled treatment on grade-specific enrollment by school location. Depending on the grade, differential mean impacts for the urban subsample turn from negative to positive or are positive and grow in size over the three years. The third-year differential mean impact in the urban subsample is 12 students in first grade, and declines monotonically by grade to 1.9 students in fifth grade. As a _percentage_ of baseline mean grade-specific enrollment in urban schools, the differential mean impacts across grades are similar in size. The third-year differential mean impacts are significant for all but fifth grade.

# **V. Conclusion**

Most evaluations of teacher incentives in low- and middle-income countries can be seen as efficacy trials. This study reports findings from an effectiveness trial in a low-income setting, where, in contrast to the efficacy trials, the government was responsible for the design and implementation of a teacher performance pay program in public primary schools, including using their own administrative data for measuring performance. Monetary bonuses provided to teachers were linked to gains in school enrollment; to gains in school mean scores in a fifthgrade standardized exam; and to the level of participation in the exam by fifth-grade students. To test whether leveraging head teachers either suffices or further augments program impacts, the receipt and size of the bonus were also randomly assigned across schools according to whether or not the teacher was the school’s head teacher.

Looking at the first three years of implementation, and using the government’s program administrative data for the evaluation, we find mixed evidence of significant mean impacts on school enrollment, exam participation rates, and student exam scores. Significant positive mean impacts are concentrated in the third year. The program raises school enrollment and exam participation rates in the third year. The trends in impacts over the three years however suggest possible learning by teachers on how to respond to the program or possible growing confidence among teachers in the administration and continuation of the program. We do not find that the program raises student exam scores in any year. Inference results for the program as a whole are mirrored by each of the treatment variants in the case of exam participation rates and mean student exam scores and by the treatment variant that offers bonuses solely to head teachers in the case of school enrollment. Post-hoc subgroup analysis reveals that the third-year program mean impacts on school enrollment and exam participation rates in the total sample are driven by 21
the urban and rural subsamples, respectively. Program mean impacts on student exam scores differ between the urban and rural subsamples in the first year but the differences in later years are not significant.

<mark>Given that the program increases exam participation rates, we test whether the absence of significant positive mean impacts on student exam scores is due to negative selection of students into examtaking. We do not find evidence supporting this hypothesis. Aside from this test, we are left to speculate on factors t</mark> hat could have blocked impacts. Some studies argue or show that implementation failures can explain the lack of program impacts.<sup>23</sup> <mark>We d</mark> iscount that the absence of positive mean impacts on student exam scores is due to this problem. Third-party validations indicate that the program has been implemented satisfactorily, and markedly better than many other programs administered by the provincial education department. Program registration and orientation, the distribution of school performance score cards, and the transfer of bonus awards to teacher bank accounts were carried fully and in a timely and efficient way. The PEC exam and ASC survey activities were also carried out as scheduled. Credibility in the program was reportedly low in the first year of implementation: program school teachers conveyed that they were initially skeptical that they would receive bonuses (consistent with program rules). This skepticism appears to have faded away after program school teachers experienced how the government performed in running the program and they received their bonus awards.

We also discount that the absence of positive mean impacts on student exam scores is due to bonus awards being too low, the bonus formula being too complex for teachers to understand, or the group-based nature of the bonuses (in the case of the two treatment variants that offer bonuses to all teachers) inducing significant freeriding. The mean bonus payout was 16 percent of the yearly basic salary for a teacher at the baseline mean pay grade and with baseline mean years of government service.<sup>24</sup> All definitions and data sources for bonus calculations were communicated by program administrators to program school teachers during school visits and at

> 23 For example, a new program may experience teething problems (see Bouguen, Filmer, Macours, and Naudeau 2013). In such cases, positive impacts may only materialize well after program inception, when problems are effectively resolved. Separately, implementation performance may be sensitive to program implementer characteristics such as the implementer’s interest, commitment, capability, and effort (Allcott 2015), and governments may be more likely or have larger shortfalls in these characteristics relative to, for example, (certain types of) NGOs (Bold et al 2013).

> 24 This relative size is several percentage points higher than, for example, the bonuses in teacher-incentive field experiments in India (Muralidharan and Sundararaman 2011) and Kenya (Glewwe et al 2010); both experimental evaluations find positive mean program impacts.

22
orientation meetings for every bonus round. Teacher bonuses are also linked to own-school performance only, and not, for example, to own-school performance compared to other schools—the latter formulation could introduce added complexity and uncertainty for teachers (Fryer 2013). The mean number of teachers in program schools at baseline was three. It is likely easier to effectively counteract any incentives for freeridership in such small schools than in larger schools with an identical bonus design (Goodman and Turner 2013).

We posit three hypotheses for the absence of positive mean impacts on student exam scores. First, the binding constraint may be external to the link between teacher effort and student exam scores. Nonschool and school factors outside the control of teachers may make improving student exam scores difficult even with greater teacher effort. Although the public schools targeted by the program were those with the lowest mean student exam scores in the province, and thus have ample room for improvement, the targeting places the program under circumstances that may be especially handicapping. In the face of these external constraints, teachers may choose to withhold greater effort or see expended effort go to waste.

Second, the binding constraint may be that teachers lack the knowhow to improve student exam scores, essentially blocking the link between teacher effort and student exam scores. Teachers may not know what exact strategies to pursue to raise student exam scores, and whether what they intend to pursue is the most efficient way to raise student exam scores. In the face of this uncertainty, teachers may withhold greater effort or misdirect their effort (Fryer 2013).

Third, teachers may optimally choose to direct their effort at those incentivized margins—school enrollment, exam participation rates, and student exam scores—that maximize payoffs net of effort costs (Neal 2011). In this case, it appears that the net payoff-maximizing margin is school enrollment in urban areas, exam participation rates in rural areas, and not student exam scores anywhere. The extent to which movement on a given margin pays off would depend on how accurately movement is measured, and teacher effort would be sensitive to this (Baker 2002). Measurement error may be higher for student scores in PEC exams than enrollment information in the ASC surveys. Notwithstanding, if measurement error is large, PEC exam scores would be a highly imperfect measure of latent student academic achievement. The latter is one of the main targets of the program.

23