metadata: authors: 0: author: Barrera-Osorio, Felipe
1: author: Raju,Dhushyanth
count: Pakistan
docna: 0: docna: Short-run learning dynamics under a
            test-based accountability system : evidence from Pakistan
docty: Policy Research Working Paper
subtopic: Secondary Education,Economics of Education,Educational Sciences,Educational Institutions & Facilities,Educational Populations,Education for Development (superceded),Education Finance,Educational Policy and Planning - Textbook,Effective Schools and Teachers,Education For All
teratopic: Education
abstracts: cdata!: Low student learning is a common finding
            in much of the developing world. This paper uses a
            relatively unique dataset of five semiannual rounds of
            standardized test data to characterize and explain the
            short-term changes in student learning. The data are
            collected as part of the quality assurance system for a
            public-private partnership program that offers public
            subsidies conditional on minimum learning levels to low-cost
            private schools in Pakistan. Apart from a large positive
            distributional shift in learning between the first two test
            rounds, the learning distributions over test rounds show
            little progress. Schools are ejected from the program if
            they fail to achieve a minimum pass rate in the test in two
            consecutive attempts, making the test high stakes. Sharp
            regression discontinuity estimates show that the threat of
            program exit on schools that barely failed the test for the
            first time induces large learning gains. The large change in
            learning between the first two test rounds is likely
            attributable to this accountability pressure given that a
            large share of new program entrants failed in the first test
            round. Schools also qualify for substantial annual teacher
            bonuses if they achieve a minimum score in a composite
            measure of student test participation and mean test score.
            Sharp regression discontinuity estimates do not show that
            the prospect of future teacher bonus rewards induces
            learning gains for schools that barely did not qualify for
            the bonus.
geo_reg_key: 517198,517212,517191
repnme: repnme: Short-run learning dynamics under a
            test-based accountability system : evidence from Pakistan
display_title: Short-run learning dynamics under a
            test-based accountability system : evidence from Pakistan
listing_relative_url: /research/2010/11/12952629/short-run-learning-dynamics-under-test-based-accountability-system-evidence-pakistan
topicv3: Teachers Management m1326119 1523,Textbooks m1326802 2129,Education m1327048 470,Secondary Education m1327058 481,Private Education m1327271 1490
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Education
no_of_pages: 74
author_keys: 0: author_key: 237905
model_extractions: WPS5465 Policy Research Working Paper 5465 Short-run Learning Dynamics under a Test-based Accountability System

# Evidence from Pakistan

_Felipe Barrera-Osorio Dhushyanth Raju_ The World Bank South Asia Region Education Unit & Human Development Network Education Unit November 2010
Policy Research Working Paper 5465

# **Abstract**

Low student learning is a common finding in much of the developing world. This paper uses a relatively unique dataset of five semiannual rounds of standardized test data to characterize and explain the short-term changes in student learning. The data are collected as part of the quality assurance system for a public-private partnership program that offers public subsidies conditional on minimum learning levels to low-cost private schools in Pakistan. Apart from a large positive distributional shift in learning between the first two test rounds, the learning distributions over test rounds show little progress. Schools are ejected from the program if they fail to achieve a minimum pass rate in the test in two consecutive attempts, making the test high stakes. Sharp regression discontinuity estimates show that the threat of program exit on schools that barely failed the test for the first time induces large learning gains. The large change in learning between the first two test rounds is likely attributable to this accountability pressure given that a large share of new program entrants failed in the first test round. Schools also qualify for substantial annual teacher bonuses if they achieve a minimum score in a composite measure of student test participation and mean test score. Sharp regression discontinuity estimates do not show that the prospect of future teacher bonus rewards induces learning gains for schools that barely did not qualify for the bonus.

This paper—a product of the Education Unit, South Asia Region; and the Education Unit, Human Development Network—is part of a larger effort of the departments to rigorously evaluate innovative government programs supported by World Bank lending operations. Policy Research Working Papers are also posted on the Web at http://econ.worldbank. org. The authors may be contacted at fbarrera@worldbank.org and draju2@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Short-run learning dynamics under a test-based accountability system: Evidence from Pakistan<sup></sup>

Felipe Barrera-Osorio<sup>†</sup> Dhushyanth Raju<sup>†</sup> _JEL classification codes_ : I21; I28 _Keywords_ : education; Pakistan; private schools; subsidies; learning; test accountability; teacher incentives; regression-discontinuity design.

>  We thank the Punjab Education Foundation, in particular, Mian Kashif Ijaz and Huma Rizvi, for extensive discussions on program and test design and implementation and assistance with the program administrative data. We also thank Amit Dar, Sofia Shakil, Huma Waheed, and the Government of Punjab School Education Department/Project Management and Implementation Unit for their encouragement and support of the research project. The findings, interpretations, and conclusions expressed herein are our own and do not necessarily represent the views of the World Bank, its Board of Directors, or any of its member countries. All remaining errors are our own.

> † World Bank, Washington, DC. Email addresses: fbarrera@worldbank.org; draju2@worldbank.org.
# **1. Introduction**

Low student learning is an important feature in Pakistan, as it is in much of the developing world (Andrabi et al. 2007; Glewwe and Kremer 2006; Hanushek and Wößmann 2007). The shortfalls relative to national curricular standards are exacerbated by the fact that a large share of children, particularly rural, female, and poor children, obtains little or no formal schooling in the country, likely seriously impairing their long-term socioeconomic prospects. In addition, in general, the factors and processes that produce real and sustained gains in student learning are not as well understood as the factors and processes that produce real and sustained gains in school participation and attainment. Evidence also shows that the factors and processes behind learning can differ from the factors and processes behind participation (Andrabi et al. 2007). This background makes the search for effective policy solutions to improve learning outcomes pressing yet tenuous.

In this study, we use five semiannual rounds of standardized test data to characterize and explain the evolution of student learning over a period which spans three academic years (2007/08-2009/10). The tests were conducted in low-cost private schools supported by public cash subsidies under an innovative test-based accountability program in the province of Punjab, Pakistan administered by the Punjab Education Foundation (PEF), called the Foundation Assisted Schools (FAS) program. The data are repeated cross-sections at the student level and longitudinal at the school level. Such data are relatively rare for a developing country. Additionally, the data are unusual for a low-income setting given that they are generated from what appears to be a well-designed and robust quality assurance testing system integrated into the administration of the program, providing a high degree of confidence that the test scores can be interpreted as sound estimates of underlying learning.

Initiated in 2005, the FAS program provides conditional cash subsidies to low-cost private schools with the objective of offering private schooling opportunities for children from low-income households and raising the level of learning in low-cost private schools. The cash subsidies are provided monthly and on a per-student basis, with essentially no conditions on how the subsidy is to be used by the program school. The subsidy amount is purposely set low to
ensure that only low-cost private schools self-select to participate in the program; the original amount was also purposely set at half the estimated per-student cost in the public school system.<sup>1</sup> In return for receiving the subsidy benefit, the program school has to, among other things, waive tuition and fees for all students and ensure that the school achieves a minimum student pass rate in the Quality Assurance Test (QAT). Program schools that satisfy the above conditions are also eligible for other substantial cash benefits offered on an annual basis: group-based bonuses for teachers in schools that achieve high QAT pass rates/mean scores and competitive bonuses for schools that rank highest in the QAT in each main program district.

Each of the above incentive features (the structure of the benefits as well as the benefit eligibility rules) represents key program innovations in the education field that may induce participation, equity, and learning gains.<sup>2</sup> We are not aware of any other program in a developing country which combines these innovations into a single intervention. As of June 2010, the FAS program has proceeded through six phases of expansion and supports 798,000 students in 1,779 schools in 29 of the 36 districts in the province, making it one of the largest PPP initiatives in education in the developing world.

The learning data used in the study come from the QAT. The QAT is a curriculum-based, multi-subject, written test offered biannually in program schools. It is designed by professional subject specialists in PEF, and, starting with QAT 4, the administration of the test was competitively outsourced to independent testing agencies. The testing agencies are expected to adhere to the testing protocols set by PEF, with testing agency compliance monitoring by PEF at all program schools. Importantly, the testing agencies are instructed by PEF to follow preestablished procedures to control for certain types of potential strategic responses of schools (i.e., cheating in a broad sense) that may artificially raise test scores.

To date, nine rounds of the QAT (QAT 1-QAT 9) have been administered, of which data from five rounds, QAT 4-QAT 8, administered between November 2007 and November 2009, are used in this study. QAT 4 is the first round of test data used in the study due to sample size considerations: it is the first QAT administered after the first major expansion of the FAS program.

> 1 Setting the subsidy level in such a way was considered important by PEF for securing political buy-in for the use of public funds for the initiative.

> 2 The typical subsidy program is designed to finance stipulated education inputs (Gauri and Vawda 2003).

3
We examine four questions. The first three questions are descriptive in nature. First, we explore how student learning at the school level has evolved in program schools over QAT rounds by examining summary statistics of the mean QAT score distributions and stochastic dominance relations between pairs of successive mean QAT score distributions. Second, we estimate the share of student-level QAT score variation attributable to different levels (district, school, grade, and student) and how the shares and levels of QAT score variation have evolved over QAT rounds. Third, we estimate the conditional mean relationship at the school level between basic pre-program school characteristics and the evolution of mean QAT scores over QAT rounds, controlling for school-level heterogeneity using alternative panel regression methods.

The fourth question is causal in nature. We estimate the causal effects at the school level of specific program benefit rules on student learning in program schools. The first subquestion we ask is whether high-powered stick incentives can induce learning gains in program schools. Program schools that fail to achieve the minimum pass rate in a given QAT are offered a second opportunity in the following QAT. If the program school fails to achieve the minimum pass rate in two consecutive attempts, the school is permanently disqualified. This creates a high-stakes situation for first-time failers to rapidly boost their learning performance in order to avoid program disqualification. Applying a sharp regression discontinuity (RD) design to the data, we examine whether accountability pressure induces gains in mean scores in the ultimatum QAT among marginal first-time failers. We also examine whether any gains found persist over postultimatum QAT rounds. Such persistence is at times interpreted in the literature as indicating that gaming behavior is unlikely to be an explanatory factor though it cannot be ruled out.

Several studies have examined the effects of (the threat of) sanctions and assistance on low-performing schools on performance in subsequent test rounds (see, e.g., Figlio and Rouse 2006; Chiang 2009; Rockoff and Turner 2008; West and Peterson 2006; and Rouse et al. 2007). Much of this research has focused on state public education systems in the United States where schools are rated on a letter grade scale from A to F based on summarizing multiple performance measures, including student performance on standardized tests, and where F-rated schools are 4
subjected to (the threat of) sanctions.<sup>3</sup> Typically applying a sharp regression discontinuity design to the data, these studies consistently find short-run gains in test scores among (students in) marginal F-rated schools in the test which is included in the determination of the rating, with the estimated impacts ranging between 0.05−0.15 standard deviations. Some of the studies additionally find that (i) test score gains persist in the medium run among (students from) marginal F-rated schools, (ii) the estimated effects are unlikely to be driven by test-taking pool selection; and (iii) “teaching to the test” is likely to be an important driver as inferred from the small or lack of effects in standardized tests that are not used in the determination of the rating.

The second subquestion we ask is whether carrot incentives can induce learning gains in program schools. Program schools qualify for substantial group-based teacher bonuses if they de facto achieve a minimum score of 100 in a composite measure of the number of students that took the QAT and the mean percentage score in the QAT. Also applying a sharp RD design to the data, we examine whether bonus nonqualification induces gains in mean scores among marginal nonqualifiers in the QAT round following the distribution of the teacher bonuses. We also examine the persistence of the carrot effect over subsequent QAT rounds. Though the carrot incentive question posed here is not perfectly correspondent (as the focus here is on the effects on nonbeneficiaries as opposed to beneficiaries), it broadly fits into the literature on the causal effects of teacher incentive schemes which directly tie teacher compensation to student learning levels and changes (Glewwe et al. 2009). To date, the limited but growing evidence is inconclusive with, for example, some studies finding positive effects on student learning (Lavy 2002, 2009; Muralidharan and Sundararaman 2009) and others finding positive effects on student learning which disappear post-intervention, suggesting temporary artificial gains in learning (Glewwe et al. 2003).

The remainder of the paper is organized as follows. Section 2 describes the education context in Pakistan at the time that the FAS program was introduced. It also describes in detail the main design and implementation features of the program. Section 3 describes in detail the design and administration features of the Quality Assurance Test as well as the data used in the study. Section 4 discusses the evolution of the school-level mean QAT scores in core subjects over QAT rounds. Section 5 discusses the student-level QAT score variance decomposition

> 3 Sanctions for F-rated schools include a combination of stigmatization from publicly disclosing ratings, higher scrutiny and oversight by school system administrations, permission/private school vouchers for students to shift out of F-rated schools, changing school management and staff, and school closure.

5
analysis and findings. Section 6 discusses the school-level QAT score regression analysis and findings. Section 7 discusses the regression-discontinuity analysis and findings on the threat effect on mean QAT scores arising from QAT failure on marginal first-time failers and the carrot effect on mean QAT scores and test participation arising from missing teacher bonus qualification on marginal bonus nonqualifiers. Section 8 summarizes the main findings and provides some concluding remarks.

# **2. Context and program features**

The Foundation Assisted Schools (FAS) program was introduced into an education landscape characterized by three defining features. One: equitable access to schooling and attainment and achievement are acute, persistent challenges. Estimates using household sample survey data for Punjab from 2004/05, the year prior to the launch of the FAS program, indicate that the school participation rate (grade 1+) of children ages 6–15 years was 65.7%—this share drops to 61.2%, 60.9%, and 48.7% when the sample is restricted to girls, children from rural households, and children from the poorest (bottom expenditure quintile) households, respectively. Conditional on any schooling (grade 1+), while 91.8% of individuals ages 17–21 years completed primary school (grade 5), only 40.7% of them completed secondary school (grade 10), with significantly lower shares for rural children and, in particular, poor children. The National Education Assessment System (NEAS) for Pakistan finds that scaled mean scores in mathematics and language assessments for a sample of grade-4 students in public schools in Punjab in 2005 were significantly less than 500, suggesting that, on average, students, obtained less than 50% of the points in the assessments (Government of Pakistan 2006). Das et al. (2006) and Andrabi et al. (2007) also find that the learning levels of grade-3 students in a selected sample of villages in Punjab measured in 2004 were far below curriculum standards in English, Urdu, and mathematics.

Two: the public sector, which is the dominant provider of education, suffers from chronic weaknesses which impair its ability to effectively address the challenges in education access, equity, and quality. To a large extent, public sector performance is hampered by weak accountability and incentive systems (Government of Pakistan 2009; Social Policy and Development Center 2003). This state-of-affairs is exemplified by recent evidence on relative teacher performance between private and public schools in rural Punjab. Andrabi et al. (2007) 6
find that, while public school teachers tend to be better paid and have higher levels of education, training, and teaching experience than their private school counterparts, the teacher absenteeism rate in public schools is almost double that in private schools (15% vs. 8%). They also find that, while private school teacher salaries are increasing in teacher _and_ student test scores and decreasing in teacher absenteeism, public school teacher salaries are largely unresponsive to these variables and are mainly a function of teacher credentials (education, training, and experience). The greater accountability in the private sector is likely due to market competitive forces which the public sector is not subject to, at least directly.

Three: in the wake of the public sector failure to adequately address these issues, the private sector has emerged as a major alternate provider of education, growing dramatically in size and reach. In particular, the rapid growth of a private sector that offers schooling opportunities to low-income households has created a major policy opportunity. Using data from 2000, Andrabi et al. (2006) find that there was an exponential increase in the number of private schools over the 1990s, with over 50% of existing private schools established on or after 1996. They also find that the birth rate of private schools in the recent period is higher in rural areas. These changes in the patterns of growth in private schools are also reflected in changes in the patterns of participation growth among students. Data from 2004/05 show that 18.7% of children ages 6–15 years were enrolled in private schools in Punjab, a 36% increase from 1998/99, with this increase largely attributable to private school participation increases among children from rural and poor households. The demand for private schooling is likely driven by the fact that fees in private schools tend to be low, accounting for a small percentage of mean annual household expenditure (Andrabi et al. 2008). It is also likely driven by the perceived higher quality of private schooling—evidence from rural Punjab shows that the learning levels in private schools are significantly higher than in public schools, even after controlling for a range of village, household, and school characteristics (Andrabi et al. 2007).

The FAS program and PEF are direct outcomes of the government’s recognition of the importance of leveraging the potential of the growing low-cost private sector in addressing access, equity, and quality issues in primary and secondary education. PEF is a publicly-funded semi-autonomous statutory organization established in 1991 by the government. It serves as the main institutional conduit for PPP programs in education in Punjab. The organization’s primary aims are to provide affordable private school opportunities to socioeconomically-disadvantaged 7
households and raise the quality of education in low-cost private schools. To these ends, it employs a variety of instruments such as providing vouchers to poor households in disadvantaged urban neighborhoods to attend selected low-cost private schools and supporting low-cost private schools with monthly per-student subsidies conditional on school quality standards, teacher training courses on subject content and pedagogical and classroom management techniques, and the insertion of competitively-hired subject specialist trainers in secondary schools for fixed terms.<sup>4</sup> The FAS program is PEF’s largest program. In the fiscal year 2009−10, PEF spent roughly 2.4 billion rupees (US$28.7 million, in current dollars) on the program—this amount accounts for 93% of total expenditures by the organization in that year. It was initiated in November 2005 on a pilot basis in 54 schools in seven districts in Punjab. Since then, PEF has rapidly expanded program coverage in phases in terms of additional districts as well as more schools within districts. The program has proceeded through six phases (the pilot phase is considered by PEF to be phase 1). Phase-6 program schools entered the program in April 2010. As of June 2010, the program covers 798 thousand students in 1,779 private schools in 29 out of the 36 districts in Punjab. As a result of explicitly targeting the program at high illiteracy districts over phases 3−5, the majority of program students and schools (70% and 68%, respectively) are in seven districts located in the southern part of the province (see Figure 1).

Table 1 provides summary statistics based on current administrative data on program schools separately by phase of entry. The sample consists of 1,776 primary (classes 1–5), middle (classes 1–8), and secondary schools (classes 1–10, 6–10).<sup>5</sup> Looking at the aggregate sample (pooled across phases), the mean program school size is 455 students; in schools with both girls and boys enrollment, the mean ratio is 1 (gender parity). Program schools have on average 17 teachers and 16 classrooms. The mean student-teacher and student classroom ratios are 27:1 and 29:1, respectively. The majority of program schools is middle level (64%), coeducational (97%), rural (59%), and registered (100%). The phase-wise statistics indicate that mean enrollment is positively associated with length of program participation, this relationship also holds for mean boys and girls enrollment. The number of teachers and classrooms are also positively associated

> 4 The organization is continuing to expand its set of instruments: for example, it is now considering supporting private organizations/individuals that adopt and manage non-functional public schools handed over by the government for this initiative.

> 5 The three higher secondary schools in the program are excluded.

8
with length of program participation, increasing commensurately with enrollment increases as inferred from the stable student-teacher and student-classroom ratios across phases.

The FAS program offers three types of cash benefits to schools. These benefits were introduced at different points in time. First, the program offers a per-student subsidy, which was introduced at program inception. The monthly per-student subsidy amount was fixed at Rs. 300 (US$3.5) for all students, which was roughly half of the estimated per-student cost in the public primary and secondary education system at the time the program was initiated. Starting in September 2008, the monthly per-student subsidy was raised to Rs. 350 (US$4.1) for students in grades 1–8 and Rs. 400 (US$4.7) for students in grades 9–12. The subsidy benefit is provided to schools on a monthly basis for all twelve months in the year. To facilitate timely and regular payments, starting in August 2007, the subsidy benefit amounts are electronically transferred to the bank accounts of program schools.

Second, first announced in December 2006, every academic year, the program offers cash bonuses to teachers in high-performing schools. Details on the eligibility criteria for the bonuses are provided in Section 7 but a maximum of five teachers in program schools that achieve (in part) a minimum student pass rate of 90% (i.e., at least 90% of tested students obtain a minimum percentage of 40%) in the QAT receive a bonus award of Rs. 10,000 (US$118) each. This is a substantial bonus amount for teachers in program schools: using available data on maximum and minimum monthly teacher salaries from applications to the program by phase-3 and phase-4 program schools, we estimate that the bonus amount represents 29% and 59% of pre-program mean annualized maximum and minimum salaries. This bonus size is approximately an order of magnitude larger than the range of sizes observed in most teacher incentive programs (see Glewwe et al. 2003). In practice, the bonuses are offered to teachers who taught the classes and/or subjects that were tested in the QAT. Similar to the subsidy benefits, the bonuses are transferred electronically to the personal bank accounts of the teacher awardees. To date, four rounds of teacher bonuses have been awarded.

Third, first announced in February 2007, every academic year, the program offers a competitive school bonus to the top-performing school in each major program district (there are seven major program districts). The program school with the highest student pass rate in the QAT is awarded Rs. 50,000 (US$588).<sup>6</sup> This bonus size is relatively modest: given a pre-

> 6 Ties between schools in pass rates are broken by looking at mean percentage scores in the QAT.

9
program mean school size of 252 students in phase-3 and phase-4 schools based on data from program applications and a per-student subsidy of Rs. 300, the bonus amount represents 6% of the expected mean annual subsidy payment to program schools. Again, the bonuses are transferred electronically to the bank accounts of the school awardees. To date, three rounds of school bonuses have been awarded.

Once schools qualify and join the FAS program, the partnership contract stipulates several conditions for maintaining benefit eligibility. The conditions that are stringently applied by PEF are (1) schooling is offered to students without charging them any tuition or fees (and displaying this status prominently on a PEF-issued signboard outside the school gate) and (2) participation of the program school in the QAT and that at least 66.67% of the tested students score 40% or higher on the QAT. A one-time violation of these conditions typically results in a warning and the capping of enrollment figures for the subsidy payment until the next QAT round. A second violation results in the permanent disqualification of the school with immediate effect. With the QAT-related condition, the school is permanently disqualified if it fails to achieve the minimum pass rate in the QAT in two consecutive attempts. Since the start of the program, there have been 51 schools that have been disqualified from the program due to double failure on the QAT.

There are also other conditions for maintaining benefit eligibility. These include (1) registering the school with the District Registration Authority within one year of joining the program; (2) conducting only one class in a classroom in any period; (3) maintaining or upgrading the quality of the school’s physical infrastructure (e.g., adequate classroom space, properly-constructed rooms and buildings, sufficient ventilation, and sufficient artificial and natural light); (4) adequate furniture and teaching tools (e.g., benches, desks, and blackboards); (5) monthly reporting on enrollment figures; (6) maximum student-teacher and studentclassroom ratios of 35:1; (7) a minimum enrollment of 100 students; and (8) no after-hours classes or tutoring services at the school. These additional conditions are applied more leniently; typically, when PEF detects a violation among this subset of conditions, schools are provided with a warning and a grace period within which to comply. To date, no program schools have been disqualified for repeated violations of these conditions.

10
# **3. The Quality Assurance Test**

The Quality Assurance Test (QAT) is the backbone of the FAS program, as eligibility for program benefits are directly tied to the performance of the school in this test. The test has several design and administration strengths. In terms of salient design features, the QAT is designed by professional subject specialists at the Academic Development Unit (ADU) in PEF. It is a criterion-referenced test based on learning standards in the national curriculum.<sup>7</sup> It tests five levels of cognitive learning under Bloom’s (1956) classification: knowledge, comprehension, application, analysis, and synthesis. The test duration varies between 45–65 minutes, with shorter test durations for lower grades. The QATs designed for students in primary and elementary grades have sections in English, Urdu (the vernacular language in Punjab), mathematics, and general science; those designed for students in secondary grades have sections in English, Urdu, mathematics, and a combination of three of the following five subjects: biology, chemistry, and physics, electrical application, and computer science (the combination of the physical science subjects is the norm).<sup>8</sup> Except in the English and Urdu sections, where the questions are matching, short-answer, or essay type (Urdu section only), questions in the rest of the subjects follow a four-option multiple-choice format. Instructions for the QAT are provided in Urdu; the questions and multiple answer choices in the non-language sections are in both Urdu and English. The QAT is offered twice every academic year, first in November and second in March. The tests are structured identically and are of equal difficulty but more material is covered in the March test, in line with the syllabi in the program schools.<sup>9</sup> To help schools familiarize themselves with the test content and format, PEF periodically shares sample test papers with program schools. PEF has also developed and periodically shares content lists which delineate expected content knowledge, separately by grade and subject.<sup>10</sup>

> 7 The majority of program schools use textbooks from the Punjab Textbook Board as they are provided free of charge through the program—these textbooks are based on the national curriculum.

> 8 Each program school is categorized into a subject group based on information provided prior to the QAT. The subject group determines the subjects tested in classes 9 and 10. The groups are Biology (physics, chemistry, biology, math, English, and Urdu), Electrical Application (physics, chemistry, electrical application, math, English, Urdu), and Computer Science (physics, chemistry, computer science, math, English, Urdu).

> 9 The November QAT tests material from the first six months of the academic year. The March QAT tests material from the full academic year.

> 10 Sample QAT papers and content lists are available on PEF’s website at http://www.pef.edu.pk/downloads-model- <u>papers.html</u> and <u>http://www.pef.edu.pk/downloads-content-list.html, respectively (Last accessed: June 15, 2010).</u> Content lists are also shared with program schools in booklet form. The first time the contents lists were shared was in June 2007, prior to QAT 4.

11
In terms of salient administration features, the testing is school-based, with students tested in their school during normal school hours. The test papers are written, formatted, sorted, counted, and sealed in envelopes by ADU. PEF also administered the tests at program schools in QAT 1−QAT 3. Starting from QAT 4, the administration of the test was competitively outsourced to independent testing agencies, namely the Board of Intermediate and Secondary Education (BISE), Bahawalpur and the Punjab University Education Testing Service, Lahore.<sup>11</sup> The testing agencies are expected to strictly follow the test administration guidelines prepared by PEF and are responsible for the administration steps starting from securely transporting the test materials to the schools and ending with hand-scoring the test following the provided scoring guides and the delivery of the student- and subject-wise test score database and filled-in and unused question papers, as well as brief test administration reports and test administration checklists for each school, to ADU. PEF is not completely divorced from test administration however—it sends a PEF staff member to be present at each test to monitor test agency compliance with the guidelines; if any problems arise, the staff member is authorized to take corrective steps. The staff member is expected to fill in and submit a monitoring report (accompanied by photographs and video footage [see Figure 2 for examples]) for each program school covered.

Although the QAT is a pre-announced test, it attempts to control for test pool selection by the school at the grade level. Prior to administering the QAT, PEF sends an intimation letter to the school some two-three weeks in advance of the planned test date, with key test planning information. The testing agency also communicates the same information to the school a few days before the planned test date, either by letter or personal visit. Importantly, schools are notified that 100% attendance is required on the test date, with attendance numbers checked against the grade-wise enrollment statement submitted to PEF for the latest subsidy benefit payment. Consistent with the test administration guidelines, in practice, the QAT is administered at the program school if the student attendance rate is at least 80% (PEF reports that test cancellation due to lower-than-required student attendance is a rare event). The number of test papers in each grade-specific packet is determined by ADU based on the latest submitted monthly enrollment statement. If the number of students in the grade selected for the QAT

> 11 PEF reportedly made the decision to outsource the testing in order to (i) increase program school confidence in the integrity of the testing process and the test results as well as (ii) release the organization from the administrative burden of performing these critical activities during a period of significant program scale-up.

12
exceeds the number of enclosed test papers, the testing agency is directed to select the longertenure students by examining the school enrollment records for the last three months (again, ADU reports that this is a rare event, as changes in enrollment usually occur at the start of the academic year and the QATs are administered well into the academic year). The testing agency also checks the enrollment and attendance records over the last three months for fictitious students and impersonation cases.

QAT procedures have been developed to discourage schools from strategically concentrating on some grades for test preparation and to limit the risk of test leakage and its consequences. In primary schools, the QAT tests two grades; in middle and secondary schools, it tests three grades. Which specific grades are selected to be tested in a given school are kept strictly confidential by the ADU team—both the testing agency and the program school learn which grades will be tested only when the sealed test packets are opened by the testing agency at the school in the presence of the school administrator(s) just before testing time. The testing agency also only brings in the sealed test packets to be used at that school on the day of the test. In addition, for each grade, eight different question papers are prepared by the ADU. These questions papers are randomized for each grade. Table 2 shows the practical results of implementing these procedures: it presents the top-four combinations of tested grades and the associated shares of program schools by QAT round.

A number of additional, more standard procedures are applied to prevent cheating. Teachers and school administers are not permitted to enter the testing area during the test. Test invigilation is typically carried out by a team of at least three persons. Children are not allowed to bring any materials or stationery (other than pens) into the testing area. The grades selected for the QAT are tested simultaneously in the same testing area and the seating for the students is mixed. All test materials provided to the test takers are collected back. All unused test materials are retained by the testing agency. ADU also reports that test item recycling across QAT rounds is presently rare given the size of its test item bank.

The QAT serves as the main source of data for the analysis of learning dynamics in program schools. While PEF has conducted nine QATs since the inception of the FAS program (QAT 9, the latest QAT, was administered in March 2010), this study uses data from QAT 4−QAT 8 which spans November 2007–November 2009, a two-year period, covering, in whole or in part, three academic years. QAT 4 is the first QAT round in the analysis as it represents the 13
first QAT after the first major expansion of the program: phase-3, which increased the number of program schools from 194 to 676 schools or by 248%.

The sample for the learning dynamics analysis is restricted to phase-3 and phase-4 program schools. This is for two reasons. First, large numbers of schools entered into the program in these phases, and these schools have been in the program for a sufficiently long period to examine short-run dynamics (phase-3 schools have been subjected to five QATs and phase-4 schools four). Second, by phase-3, PEF had formalized the program entry qualification process, introducing standardized application forms, unannounced school inspections by PEF staff with standardized inspection forms, and an entry test, called the Short-Listing Quality Assurance Test (SLQAT), which is a pared-down version of the QAT. As a result, pre-program information on schools from the entry qualification stages is available for phase-3 and phase-4 schools, allowing the examination of the correlations between pre-program characteristics and the learning levels and changes, as well as their use as a form of specification testing in the causal analysis of the effects on learning from program design features that introduce stick and carrot incentives.

The data were provided by PEF in QAT-specific databases. Each QAT database is at the student level, with the number of points scored and/or the percent score in each relevant subject. Information on the school and grade of the tested student is also provided. Schools in the databases for QAT 5−QAT 8 were identified by both their names and unique identification numbers provided by PEF. School-level summary statistics on QAT performance—such as the number of test takers, test score means, test score standard deviations, and test pass rates— derived from the student-level data were linked across QAT rounds using the unique school identifiers. There were no issues in the linking process: when a school in a given QAT was expected to be also found in another QAT, it was. Program schools in the QAT 4 database were however only identified by school name (at times augmented by location markers inserted into the school name field). Thus, schools in QAT 4 were linked to the schools in QAT 5 by visually matching on school names. There were 674 program schools that took QAT 4; 658 of them were expected to be found in QAT 5. Matching on school names yielded 625 linked schools (a success rate of 95%).

Table 3 presents the number of phase-3 and phase-4 program schools that took each of the QATs. It also presents the number of phase-3 and phase-4 schools that qualified for entry and 14
joined the program. The first QAT that phase-3 program schools took was QAT 4 in November 2007. The first QAT that phase-4 program schools took was QAT 5 in March 2008. It appears that until QAT 8, there was virtually no school attrition out of the program. Even in QAT 8, the attrition was minor: only 3% of phase-3 and phase-4 program schools exited the program. The majority of schools that exited did so due to disqualification for consecutive failures on the QAT. This finding of low attrition from the program indicates that sample selection of this form is unlikely to be a source of bias in the learning dynamics analysis.

Table 4 presents the number of students tested in the various QAT rounds, separately by grade. The sample comprises of all program schools. We discern two patterns that signal test design and administration procedures at work. First, the number of grades that are potential targets for testing has expanded over QAT rounds: in QAT 4, the tested grades were drawn from a universe of four grades. By QAT 8, it was nine. Second, the changing shape of the frequency distribution of test takers by grade between QAT rounds is consistent with the procedure of changing the combination of grades tested in a given program school over QAT rounds.

# **4. Evolution in learning**

In this section, we examine how learning has evolved over QAT rounds among phase-3 and phase-4 program schools, where learning is measured in terms of mean QAT scores in the core subjects of English, mathematics, and Urdu. Table 5 presents means and standard deviations for school-level mean QAT scores by QAT round. These statistics are presented for total mean QAT scores as well as the subject-specific mean QAT scores. Figures 3 and 4 depict kernel estimates of the probability density functions (PDFs) for total mean scores by QAT round for phase-3 and phase-4 program schools, respectively. Mean QAT scores for both phase-3 and phase-4 programs are normalized using the mean QAT 4 score distribution for phase-3 program schools as the base.

Looking first at phase-3 program schools, we find that mean scores jumped 2.90 standard deviations between QAT 4 and QAT 5. Figure 3 clearly displays this distributional shift. This is a massive change that occurred over a short period of four months. In Section 7, we examine whether this substantial change in learning is attributable to some extent to accountability pressure on a large number of phase-3 program schools that failed to achieve the minimum pass rate for program benefit maintenance in QAT 4. Changes in learning after QAT 5 are relatively 15
more modest, with mean scores backtracking somewhat in QAT 7 and QAT 8 relative to those of QAT 6 by 0.42 and 0.36 standard deviations, respectively. Decomposing the mean total score into its constituent subjects, we find that the mean scores in all subjects depict the same major increase in learning between QAT 4 and QAT 5 as the mean total score. Mean English and mathematics scores continue to show similar trends over QAT rounds as the mean total scores; mean Urdu scores however appear to follow a counter-trend to the mean total scores in later QAT rounds. Similar to the trend for the level of mean scores, the _variation_ in mean scores as measured by standard deviations shows a pattern of increase (peaking in QAT 6) and then decline over QAT rounds. Variations in mean English and mathematics scores depict similar trends to the variation in mean total scores, while variation in mean Urdu scores follows a counter-trend to the variation in mean total scores.

Phase-4 program schools entered in the program prior to QAT 5. Mean scores in QAT 5 were 2.51 standard deviations higher than the mean scores for phase-3 program schools in their first QAT, QAT 4. However, the mean scores were 0.39 standard deviations lower than how phase-3 program schools performed in QAT 5. Over QAT 5−8, mean scores for phase-4 program schools do not show any discernible trend, with mean scores fluctuating between 2.5 and 2.9 standard deviations. Similarly, the subject-specific mean scores show no discernible trend, displaying stronger fluctuations over QAT rounds. The effects of these fluctuations on mean total scores are however dampened by the counter-fluctuation of mean Urdu scores vis-à-vis mean English and mathematics scores. Variation as measured by standard deviations in mean total scores shows an increase from QAT 5 to QAT 6 and then a decline over subsequent QAT rounds; similar trends are exhibited by the subject-specific mean scores.

In addition to examining summary statistics of the mean QAT score distributions, we test for stochastic dominance relations between successive pairs of mean QAT score distributions for phase-3 and phase-4 program schools separately. Suppose we have two distributions _A_ and _B_ , characterized by their cumulative distribution functions (CDFs) _FA_ and _FB_ , respectively. The distribution _B_ stochastically dominates distribution _A_ at first order, if for any argument _z_ , _FA_  _z_  _FB_  _z_ . If _Z_ denotes learning and the observations are schools, this implies that the incidence of schools with learning values equal to or less than a ‘low’ learning level _z_ is higher under distribution _A_ than under distribution _B_ . Higher orders of stochastic dominance can be 16
_s_ defined analogously. Denote _Dk_ to be dominance function of order _s_ for distribution _K_  _A_ , _B_ , defined recursively by the relations Distribution _B_ stochastically dominates distribution A at order _s_ if _DAs_  _z_  _DBs_ for all values of _z_ in the joint support of the two distributions. Stochastic dominance of the second order implies that the low learning gap is higher under distribution _A_ than under distribution _B_ . Stochastic dominance of the third order implies that the squared low learning gap is higher under distribution _A_ than under distribution _B_ . We test for stochastic dominance based on an empirical likelihood ratio statistic proposed by Davidson and Duclos (2007) and extended by Davidson (2009).<sup>12</sup> This test tests for the rejection of the null hypothesis of nondominance between distributions _A_ and _B_ . The test tests first order as well as higher order stochastic dominance relations and permits the distributions under examination to be correlated, which is likely given serial correlation in the mean QAT score distributions. The test is restricted to the mean QAT score values between the 5<sup>th</sup> and 95<sup>th</sup> percentiles of the joint support of the two distributions.

Figures 5 and 6 present CDFs of the mean QAT scores by QAT round for phase-3 and phase-4 program schools, respectively. For phase-3 program schools, pair-wise tests suggest that the distribution of mean QAT scores for QAT 5 and subsequent ones first-order dominate the corresponding distribution for QAT 4. There is no evidence of stochastic dominance up to the third order between successive mean score distributions among QAT 5−QAT 8. Pair-wise tests of the successive mean score distributions among QAT 5−QAT 8 for phase-4 program schools also suggest that the null hypothesis of nondominance up to the third order cannot be rejected under standard significance levels.

Thus the evidence collectively points to two main findings. First, there was a major jump in learning as measured by mean scores between QAT 4 and QAT 5 for phase-3 program schools. Second, between QAT 5−QAT 8 learning appears to be in an oscillating steady state over QAT rounds for both phase-3 and phase-4 program schools. This finding remains valid when mean total scores are decomposed into mean scores for English, mathematics, and Urdu.

> 12 The test is automated in `Stata` by Araar and Duclos (2009).

17
# **5. Learning variance decomposition**

In this section, we examine how much of the total variability in QAT scores is attributable to the variability in QAT scores _between_ program schools (i.e., due to school heterogeneity) versus the variability in QAT scores between students _within_ program schools (i.e., due to student heterogeneity). Decomposing further, we examine how much of the total variability in QAT scores is attributable to the variability in QAT scores (1) between program districts, (2) between schools within district, (3) between grades within school, and (4) between students within grade. These are important questions as education policymakers and practitioners may have at their disposal the means to effectively reduce the extent of heterogeneity at the level of the school and grade (if they are found to be major contributing factors), thereby reducing total variability in student achievement.

To answer the above questions, we fit two-level and four-level variance-components models (nested random effects models) to the student-level QAT score data. This exercise is performed separately for each round of QAT score data.<sup>13</sup> It is also performed separately for phase-3 and phase-4 program schools.

The two-level variance-components model is specified as where _i_ indexes the student and _s_ the school, and<sup>_y_</sup> _is_<sup>denotes the mean QAT score in the core</sup> subjects of English, mathematics, and Urdu for student _i_ in school _s_ . Students are nested within schools—thus student QAT scores within schools are likely to be correlated. The QAT sometimes tested students in the same grades across schools. As a result, the grade _g_ is viewed as a cross factor with the school _s_ and is treated in the model as a fixed vector of grade dummies<sup>_x_</sup> _g_ with a vector of fixed parameters  to be estimated. The error components  _s_ and  _is_ are assumed to be independently normally distributed with means zero and standard deviations  _s_ and  , respectively, and are estimated correcting for potential correlation among students within schools and heteroskedasticity of arbitrary form. Since the error components (  _s_ and  _is_ )

> 13 We are unable to pool the student-level QAT score data over QAT rounds in order to examine how much the variation in student QAT scores is explained by inter-round variation, as a panel only exists at the school level; at the student level, we have repeated cross-sections.

18
are assumed to be independently distributed, the total variance in QAT scores _Var_  _yis_  is equal to the sum of the between-school variance  2 _s_  and the within school, between-student 2 variance   .

The four-level variance components model is specified as where _i_ indexes the student, _g_ the grade, _s_ the school, and _d_ the district, and _yigsd_ denotes the mean QAT score in core subjects for student _i_ in grade _g_ in school _s_ in district _d_ . There are four factors in this model: district, school, grade, and students. Students are nested within grades, grades within schools, and, in turn, schools within districts. These factors are treated as additive random effects in the model, where  _d_ ,  _sd_ , and  _gsd_ denote the random effects varying over district _d_ , school _s_ , and grade _g_ , respectively. These random effects are assumed to be independently and normally distributed with means zero and standard deviations  _d_ ,  _sd_ , and  _gsd_ , respectively, and are estimated correcting for potential correlation among students within schools and heteroskedasticity of arbitrary form. As in (1), the grade _g_ is additionally treated in the model as a fixed vector of grade dummies<sup>_x_</sup> _g_<sup>with a vector of fixed parametersto be</sup> estimated. Given that the error components are assumed independent, total QAT score variation is equal to the sum of the variances of the error components. Both variance-components models are estimated via maximum likelihood.

The results of the QAT score variance decomposition are presented in Table 6. Panel 1 presents the estimated shares of total variation at the selected levels for phase-3 FAS program schools, separately by QAT round. Panel 2 presents analogous results for phase-4 FAS program schools. We highlight four main results. First, the results in general suggest that most of the variation in QAT scores is at the level of the student—for example, the two-level variance components estimation suggests that roughly 60–70% of total QAT score variation is between students within schools, while the remaining 30–40% is between schools. This finding does not change qualitatively when we look at the results from the four-level variance-components estimation: though the share falls to 50–60%, the variation in QAT scores between students remains the majority contributor to total QAT score variation.

19
Second, the four-level variance-components estimation suggests that the variation in QAT scores between program districts explains a small share of the total variation in QAT scores: between roughly 1–5%. This finding implies that the QAT score distributions are essentially identically centered across districts **.** Third, the estimation also suggests that within schools, the variation in QAT scores between tested grades explains a nontrivial share of total QAT score variation (around 15–18%). In addition, the share of total variation due to betweengrade variation appears to be drawn from the shares due to between-school variation and between-student variation—this absorption from both directions likely arises from the fact that tested grades are not only nested within schools but are also crossed with schools, given that sometimes students in the same grades are tested across schools.

Fourth, the estimated total variation in student scores shows substantive movements over QAT rounds. Figure 7 depicts the evolution of the _levels_ of total, between-school, and withinschool variation in QAT scores, separately for students in phase-3 and phase-4 schools. In phase3 program schools, the level of total student score variation declines from QAT 4 to QAT 5 before rising sharply with QAT 6, and then declines monotonically over QAT 7 and QAT 8. The level of total student score variation for phase-4 program schools shows an identical trend, with a sharp increase between QAT 5 and QAT 6 before declining over the subsequent two rounds. It appears that the evolution in the level of total student score variation between QAT 4–QAT 6 in phase 3 schools is largely explained by the evolution in the level of within-school student score variation; between-school student score variation shows little movement over these QAT rounds.

Notwithstanding, what explains the uptick in total score variation between QAT 5 and QAT 6? There were no major changes in program design or implementation between those two rounds. The one change that did occur was the reduction in the number of QAT questions and points in the English, mathematics, and science sections from 15 to 10, while the points for the single question in the Urdu section remained fixed at 10. This test structure has been maintained through the subsequent QAT rounds. This change implies that the Urdu section now receives a higher weight in the QAT—if the student only answered the Urdu question correctly in QAT 5, she would receive 18.2% on the test; if the same event occurred in QAT 6, she would receive 25% on the test.

Given the reweighting of the QAT across subjects, the increase in total QAT score variation between QAT 5 and QAT 6 suggests that the variation in Urdu scores is higher than in 20
the other core subjects of English and mathematics. Figure 8 depicts the evolution of the level of QAT score variation by subject. While the level of QAT score variation in Urdu is not always higher than those for the other subjects, there was a major spike in the level of its variation between QAT 5 and QAT 6: Urdu score variation increased by 61% and 57% for phase-3 and phase-4 program schools, respectively. Thus, it appears that the score reweighting amplified the effect of the spike in Urdu score variation on total QAT score variation. We however cannot provide an explanation for the observed spike in Urdu score variation.

How do these test score variance decomposition estimates compare to estimates from other recent studies from the South Asia region? Our estimates of inter-school differences appear to lie on the lower end of the range of available estimates. For example, similar to our findings, on the lower end, Goyal and Saiens (2009), using 2007 data from a nationally-representative sample of grade-2 and grade-4 students from Bhutan, find that 26–41% of the total variation in test scores across tested grades and subjects is explained by variation between schools. Andrabi et al. (2007) find that, in a sample of grade-3 students in private and public schools in rural Punjab, Pakistan, roughly 50% of total variation is explained by variation between schools. On the higher end, although the statistic is not reported, Saiens (2009) finds that, in a sample of public schools from rural Sindh, Pakistan, the majority of the variation in test scores for grade-3 and grade-5 students is explained by variation between schools.

# **6. Pre-program correlates of learning dynamics**

In this section, we examine what factors are associated with learning levels and changes in program schools by essentially estimating school effectiveness regressions. The factors that we examine are basic school-level characteristics captured at the program application stage (i.e., preprogram) for phase-3 and phase-4 program schools such as school size, number of teachers, maximum and minimum teacher salaries, number of classrooms, level, gender type, location, registration status, and mean SLQAT score. Learning is measured in terms of the school-level mean QAT score in the core subjects of mathematics, English, and Urdu. The conditional relationships between these factors and learning (changes) are investigated via simple pooled ordinary least squares (OLS) regression as well as via fixed-effects (FE) and random-effects (RE) regressions which take advantage of the panel structure of the data (multiple test rounds over schools) to control for heterogeneity at the school level.

21
The pooled OLS regression model is formulated as where _s_ indexes the school and _q_ the QAT round, and _ysq_ is the normalized mean QAT score in the core subjects for school _s_ in QAT _q_ , _QATq_ and<sup>_x_</sup> _s_<sup>are vectors of the QAT rounds and the</sup> time-invariant pre-program covariates, respectively; the associated parameter vectors  and  to be estimated capture the main effects of the covariates on mean QAT scores. The vector _QATq_  _xs_ captures interactions between the pre-program covariates and the QAT rounds; the associated parameter vector  to be estimated captures the differential effects of the preprogram covariates over QAT rounds. The stochastic error term  _sq_ is assumed to be uncorrelated with the covariates and distributed normally. The classical assumption of identically and independently distributed errors is however relaxed. The error term is likely to be correlated over QAT rounds for a given school; hence, we estimate standard errors corrected for potential serial correlation as well as cross-sectional heteroskadasticity of arbitrary form. If the model is correctly specified and the error term assumptions are valid, the pooled OLS estimator is consistent.

The pooled OLS estimator is inconsistent if the true model is the fixed effects model. The fixed-effects (FE) regression model is formulated as _ysq_   _s_  _QATq_  _QATq_  _xs_    _sq_ , _s_  1,..., _S_ , _q_  1,..., _Q_ , (4) where  _s_ is a vector of random school-specific variables treated as unknown parameters to be estimated that are potentially correlated with the other regressors _QATq_ and<sup>_x_</sup> _s_<sup>. The stochastic</sup> error term  _sq_ is assumed to be uncorrelated with the covariates. The estimates of the parameter vectors  and  reflect _within_ -school effects of the QAT rounds and the pre-program covariates over QAT rounds on mean QAT scores (relative to the base QAT, QAT 4), respectively. As with the OLS standard errors, the FE standard errors are adjusted for serial correlation and crosssectional heteroskedasticity of arbitrary form.

If the true model is a random effects model, the pooled OLS estimator and the FE estimator are consistent but inefficient. The random-effects (RE) regression model is formulated as 22
where  _s_ are assumed to be random intercepts that are distributed independently of the covariates. The random intercepts  _s_ and the stochastic error term  _sq_ are assumed to be distributed normally; they are however not assumed to be independently and identically distributed.

The FE estimator is likely to be less biased than the pooled OLS estimator as the school fixed effects control for time-invariant unobserved factors that vary at the school level or higher. The model also allows for the school fixed effects to be correlated with the other covariates. However, as the estimator ignores the significant between-school variation present, the parameter estimates are likely to be imprecise. In addition, the independent effects of the timeinvariant pre-program covariates cannot be estimated and are excluded from the model as they are perfectly collinear with the school fixed effects. The RE estimator allows the independent effects of the time-invariant pre-program covariates to be estimated. However, the RE model assumes that the school random effects are uncorrelated with the covariates, which is usually unrealistic in most applications. We use the artificial regression approach to test the null hypothesis that the more efficient RE estimator yields similar parameter estimates to the consistent FE estimator (Wooldridge 2002; Baltagi 2005). This test is robust version of the standard Hausman test, accounting for cross-sectional conditional heteroskedasticity and serial correlation of arbitrary form.<sup>14</sup> As discussed in Section 4, given that significant changes in learning were only observed with phase-3 program schools, we only present results from estimating school-level learning growth-curve regressions for this sample over QAT 4–8. Table 7 presents the learning growthcurve regression results. The dependent variable in the models is the mean QAT score in the core subjects normalized using the QAT 4 score distribution. All regressions include district dummies although their parameter estimates are not reported. We find that the parameter estimates for preprogram characteristics appear to be of comparable sizes across the three estimation models. Estimates of the standard errors also appear to be similar across models; this implies that

> 14 The test is automated into `Stata` by Schaffer and Stillman (2010).

23
statistical inference is largely identical across models. Hence, we discuss the estimation results without differentiating by model.

Learning in QAT 5–QAT 8 was higher than in QAT 4. The effects largely vary between 4–5 standard deviations at the conditional mean. This evidence is consistent with the unconditional evidence on changes in learning presented in Section 4. While the main effect lacks significance, middle schools have lower learning levels than primary schools in QAT 7 and QAT 8 relative to QAT 4. Similarly, while the main effect lacks significance, secondary schools have lower learning levels than primary schools in all rounds relative to QAT 4. The main and interactions effects for both middle and secondary schools are jointly significant. Registration status of the school has a positive effect on learning but this effect disappears in QAT 6 and QAT 8. While the main effect lacks significance, program school size appears to have a small positive effect on learning in QAT 5. The number of teachers also has a small positive effect on learning. Given that we control for school size, this effect can be interpreted as the effect of lower studentteacher ratios on learning. However, the effect disappears in QAT 5–8. Neither the location of the program school in terms of rural versus urban nor the gender status of the school in terms of coeducational versus single-sex appear to matter for learning levels or changes. Pass rates in the program entry test is positively associated with learning; the effect however does not vary with QAT round. Finally, maximum and minimum teacher salaries, which taken together may be interpreted as reflecting the level of teacher quality in the program school, are not associated with learning levels or changes.

Explaining changes in learning is effectively tantamount to explaining the jump in learning between QAT 4 and QAT 5, as post-QAT 5, the learning distributions of phase-3 program schools appear to show some relatively small oscillating movement. The same situation applies to phase-4 program schools: mean scores appear to show only slight oscillating movement over QAT rounds. Tables A1 and A2 present learning growth-curve regression results over QAT 5–QAT 8 for phase-3 and phase-4 program schools, respectively. The explanatory power of the models estimated over these samples drops precipitously.

All the estimations presented above do not attempt to correct for potential measurement error in the regression covariates. Measurement error in covariates such as location, level, gender type, and registration status is unlikely but schools may overreport such characteristics as enrollment, teachers, and classrooms in the program application forms if they feel that it may 24
increase the likelihood of program entry. However, the fact that PEF visits all applicants and checks all reported figures is likely to help arrest this tendency. Notwithstanding, if present, the extent of overreporting is likely decreasing with true enrollment, teachers, and classrooms. For example, given that only private schools with at least 100 students can apply to the program, small schools may inflate the enrollment to exceed and distance themselves from this floor. If the observed values of a single covariate are inflated with measurement error negatively correlated with true values, the associated parameter estimate is upwardly biased. In a multiple regression framework, systematic measurement error of this form among multiple covariates is likely to bias regression parameters in unknown ways.

# **7. Causal effects of stick and carrot incentives on learning**

Exploiting the design feature that program assignment was ultimately strictly determined on the basis of SLQAT pass rates relative to a distinct, known cutoff and applying a sharp regressiondiscontinuity design to the data for phase-4 program schools, Barrera-Osorio and Raju (2009) find that the FAS program produced sizeable, positive effects on school size, school inputs such as teachers, classrooms, and blackboards, and student achievement in program schools with SLQAT pass rates near the cutoff. The documented learning gains induced by the program are theoretically attributable to several drivers. For example, a first potential mechanism is the QAT pass rate floor for continued program eligibility which is likely to create a strong push from below for the program school to invest resources, organize itself, and set and activate its own internal incentives to ensure that the majority of students learn enough to regularly pass the QAT. The effort exerted may also be dynamically continuous: more risk-averse program schools are likely to continue to exert an effort to raise the QAT performance of their students in order to extend the distance between their pass rate positions and the QAT pass rate floor, creating further “breathing room”.

A second potential mechanism is competition-driven pressure exerted by parents and other local stakeholders on the program school to raise student learning created by the FAS program requirement initiated with QAT 6 in November 2008 that the program school prominently publicly display the QAT pass rate performance rankings of all the program schools in the school’s district. The first and second mechanisms are likely to produce learning gains by compressing the QAT distribution from the left.

25
A third potential mechanism is the offer of annual cash bonuses to teachers in program schools that (first) achieve a QAT pass rate of at least 90% (and, second, pass another threshold based on a simple addition of the number of testtakers and mean QAT score), as well as the competitive annual cash bonus to the program school with the highest QAT pass rate<sup>15</sup> in the major program districts. Both of these bonuses—which are substantial in both absolute terms and relative to pre-program school revenue and teacher salary estimates—are likely to create a pull from above for schools to make an effort to continue to raise their QAT performance. This mechanism is likely to produce learning gains by shifting as well as stretching the QAT score distribution to the right.

A fourth potential mechanism is the per-student subsidy benefit itself, which may provide higher revenues to the program school if pre-program tuition rates were lower than the subsidy benefit and/or school size has expanded. The regular, full receipt of the per-student subsidy benefit (rather than reliance on tuition payments from parents with low and/or unstable paying capacity) also raises the program school’s confidence in the predictability of future cash flow. Both these factors may facilitate larger and longer-term investments in resources to improve teaching and learning at the school, and, hence, produce learning gains which are realized across the QAT score distribution.

A fifth potential mechanism is the tuition-free schooling condition for program benefit eligibility. Household expenditures previously allocated for tuition and fees are freed up for other household consumption and investment priorities. One potential outcome is the reduction in the opportunity cost of child time, allowing the student to attend school more regularly, stay for the full school hours, and spend after-school hours on homework assignments. Another potential outcome is households redirect (a portion of) the freed-up resources to investments and expenditures that may directly or indirectly improve student learning such as better nutrition, uniforms, books and stationery, transportation to school, and tutoring services.

The above potential mechanisms can generate real gains in student learning. However, the observed gains could also be the product of gaming behavior by schools, resulting in artificial gains in QAT scores. For example, program schools may respond to the QAT-related requirement for program continuation and additional benefits by systematically dropping lowperforming students and/or screening in and admitting high potential or high-performing

> 15 With ties broken by using mean QAT scores.

26
students, cheating on the QAT, manipulating which students are present for the QAT (which is announced in advance) and testing conditions, and devoting resources and directing staff and student effort to QAT-taking strategies as well as reorienting teaching and learning towards the subject and content matter covered in the QAT, at the possible expense of a more extensive and richer teaching and learning agenda (see, e.g., Jacob 2005 and Jacob 2007 for reviews).

These mechanisms likely work simultaneously to produce the observed learning gains under the program. Decomposing their individual contributions is likely to be difficult without structural modeling; and, even then, the data may not be sufficiently rich to permit the identification of the (full set) of structural parameters. However, it is still possible to partially unpack the black box of learning-gains production under the program via reduced-form estimation. This opportunity emerges from the combination of peculiar circumstance and program construction which allow the credible independent identification and estimation of the average causal effects of two mechanisms on defined subsets of program schools: (1) the threat of program exit for program schools that just failed the QAT a first time and (2) the prospect of receiving teacher bonuses in the next award round for program schools that just missed acquiring bonus eligibility in the preceding award round.

# _A. Threat effect on learning_

As mentioned, design and circumstance combine to yield an opportunity and empirical strategy to investigate whether accountability pressure under the FAS program induces student learning gains. First, in terms of the enabling design, FAS program schools have to maintain a minimum QAT pass rate of 67% for continued program eligibility. Schools that fail to pass a given QAT are offered a second chance to pass the QAT the next time it is administered. In the intervening period, PEF may impose additional rules and regulations on first-time failers such as freezing their enrollment counts for benefit amount calculations. The school is permanently disqualified from the program if it fails the QAT two consecutive times. This appears to be a serious turn of event: Observations reported by PEF suggest that a disqualified school is significantly worse off relative to a low-cost private school that had remained continuously outside the program, with disqualified schools experiencing student and staff flight and closure. Thus, the high-stakes nature of the QAT introduces powerful incentives for first-time failers to boost learning in order to avoid the threat of program exit.

27
In terms of circumstance, in phase 3 of the program’s expansion (the first major expansion of the program in terms of the number of schools enrolled), 514 low-cost private schools who applied to the program attained the minimum pass rate in the SLQAT, the final qualification step for program entry. Out of these schools, 482 schools (93%) signed their program participation agreements with PEF in July-August 2007 and began receiving the monthly subsidy benefit. In November 2007, three to four months after phase-3 program schools entered the program, QAT 4 (the first QAT for phase-3 program schools) was administered by PEF in all program schools. Out of 479 phase-3 program schools that took QAT 4, only 234 (49%) passed it. The high failure rate was unanticipated by PEF—such a rate had not been observed in preceding QAT rounds nor has it been repeated to date in subsequent QAT rounds. For example, in phase 4, the next major program expansion, of the 425 phase-4 program schools that took QAT 5 (which represented the first QAT for phase-4 program schools), 412 (97%) passed it.

The precise QAT pass rate cutoff for determining program eligibility combined with the unusually high failure rate among phase-3 program schools in QAT 4 provides a data design with adequate sample size to investigate whether the threat of program exit precipitated by failing QAT 4 has a causal effect on mean scores in QAT 5 (the ultimatum or threat QAT) as well as later QAT rounds (indicating effect persistence) for first-time failers near the cutoff among phase-3 program schools. Specifically, we fit a sharp RD design to the data to essentially compare mean scores in subsequent QAT rounds of phase-3 program schools that just failed QAT 4 (referred to as marginal failers), and, hence become subject to the threat, to those of corresponding schools that just passed QAT 4 (referred to as marginal passers).

To briefly delineate the identification strategy, let<sup>_y_</sup> _s_<sup>denote the mean QAT score in the</sup> core subjects of Urdu, English, and mathematics in phase-3 program school _s_ , and let the indicator variable<sup>_d_</sup> _s_<sup>denote treatment (threat) assignment, where one denotes that the school</sup> failed QAT 4, and zero otherwise. In addition, let _y_ 0 _s_ and _y_ 1 _s_ denote the potential outcomes of school _s_ in the untreated and treated states, respectively. Treatment status is assigned based on the decision rule 28
where<sup>_z_</sup> _s_<sup>denotes school</sup><sup>_s_’s QAT 4 pass rate which is perfectly observed (</sup><sup>_z_is more generally</sup> referred to as the assignment variable), _c_ the known, distinct pass rate cutoff of 67% (more precisely, 66.67%), and 1 an indicator function. Thus, the conditional probability of treatment as a function of the assignment variable _z_ , Pr  _d s_  1 _zs_  _c_ <sup>, dives discontinuously from one to</sup> zero at the cutoff, yielding a sharp RD design (Trochim 1984).

Let _e_ denote an arbitrarily small number. Under the assumption that (i) the zero-limit of the conditional expectation of the counterfactual untreated outcome for treated schools is well defined, (ii) the conditional expectation of the outcome variable exhibits local smoothness at the cutoff in the absence of the treatment, (iii) and the density of the assignment variable _z_ is positive in the neighborhood of the cutoff, the difference in mean outcomes between marginal passers and marginal failers identifies which represents the average treatment effect of the treated (ATT) or threat effect at the cutoff for phase-3 program schools (Hahn et al. 2001, Todd 2006). A consistent estimate of the threat effect at the cutoff is given by ˆ ˆ where  _y_<sup></sup> and  _y_<sup></sup> denote the conditional expectations of the outcome variable _y_ at the cutoff from below and above, respectively.

Given that we are interested in flexibly estimating the treatment effect at a single point using observations in its neighborhood, an attractive method is local smoothing using nonparametric regression. We use local linear regression (a local polynomial of order one) to individually estimate the two conditional expectations at the cutoff in (5); the selection of this estimator is motivated by its faster convergence rates at boundaries relative to standard kernel regression (Fan and Gijbels 1996; Porter 2002). The practical implementation of local linear regression requires the specification of the kernel _Kh_  , the weighting function, and the bandwidth _h_ , the window width in which the kernel function is applied. We select the triangular kernel given that it is boundary optimal and thus well suited to regression-discontinuity designs (Cheng et al. 1997). We select the optimal bandwidth _h_ by applying the plug-in method developed by Imbens and Kalyanaraman (henceforth, I-K) (2009) specifically for use in 29
regression-discontinuity settings, with _h_ determined by minimizing the mean squared error (MSE) using observations only around the cutoff.<sup>16</sup> Following the guidance in Imbens and Wooldridge (2009), we also examine the sensitivity of our inference results to setting the bandwidth to half and twice the optimal bandwidth generated from the I-K method for each of our estimations.

The local estimation via RD aids in controlling for phenomena/events that may have biased our results if we had opted for a more global estimation method. We note three potential sources. First, it is conceivable that some program schools may place in the bottom of the score distribution in a given QAT round due to a transitory bad draw but can expect to make gains towards the mean in subsequent QAT rounds (Kane and Staiger 2002). Thus, the initial mean QAT score of a low-performing school may be a misleading measure of its true QAT performance. This implies that, in our case, the learning gains experienced by first-time failers may be attributable in part to both accountability pressure and regression to the mean. However, any potential mean reversion tendency is expected to be locally smooth at the pass rate cutoff (Chay et al. 2005). Second, the estimated RD effects are net of any natural student learning depreciation that may affect gains in mean scores (Andrabi et al. 2009), with the conditional mean level (and the rate) of learning depreciation expected to be locally smooth at the pass rate cutoff. Third, PEF responded to the high QAT 4 failure rate by organizing training in January 2008 on QAT content and format and targeting it at QAT 4 failers _as well as_ marginal passers. Given this, while the training may have shifted the levels of the learning regression functions, the effect of the training on mean learning is expected to be locally smooth at the cutoff. Thus, the estimated RD effects at the pass rate cutoff net out any bias due to these sources and continue to identify the threat effect on marginal first-time failers.

We first discuss the findings from some basic model specification tests. First, we check if there is a positive mass of school observations in the neighborhood of the QAT 4 pass rate cutoff, given that this is a necessary condition for the identification strategy. This condition is verified by a visual inspection of the frequency distribution of phase-3 program schools by QAT 4 pass rate (see Figure 9). Second, although the following check does not necessarily pose a threat to identification, using Figure 9, we also visually inspect whether there is an unusual discontinuity in the density function of the QAT 4 pass rates at the cutoff, as it may indicate

> 16 The I-K bandwidth selection algorithm is automated in `Stata` by Fuji et al. (2009).

30
strategic one-way manipulation of pass rates at the cutoff (McCrary 2008). The inspection suggests that the QAT 4 pass rate density function appears to be generally naturally characterized by a jagged structure; furthermore, this structure does not appear to be unique to the QAT 4 pass rate density for phase-3 program schools as other QAT rounds exhibit a similar pattern. Third, the RD analysis is performed on schools for which data could be linked across QAT rounds. As discussed in Section 3, the success rate in linking schools was not perfect, particularly between QAT 4 and QAT 5, and the loss of the unlinked schools may serve as a source of potential sample selection bias in the RD analysis. However, a comparison between the frequency distribution of all phase-3 program schools by QAT 4 pass rate with that of only linked phase-3 program schools (Figures 9 and 10), provides no telltale signs of dissimilarities, suggesting that the problem is negligible.

Fourth, we test for local smoothness in the conditional expectation of pre-program covariates at the QAT 4 pass rate cutoff using data from submitted program application forms. Although consistent evidence of discontinuities in the conditional means at the cutoff for these covariates does not necessarily undermine the identification strategy, it does cast doubt on its plausibility. Table 8 presents summary statistics on the pre-program number of students, number of teachers, maximum monthly teacher salaries, number of classrooms, gender, location, level, registration status, and mean SLQAT scores, as well as the RD estimates of the effects at the cutoff using local linear regressions. The findings strongly suggest that the null hypothesis of local regression smoothness in conditional expectations at the cutoff cannot be rejected at standard significance levels for the full set of pre-program covariates examined.

Fifth, we test if the conditional mean pre-program outcome (mean QAT 4 scores) exhibits local smoothness at the cutoff. This directly tests a necessary condition for identification under RD. Table 9 shows that the RD estimate of the threat effect on normalized mean QAT 4 scores at the cutoff are is −0.003 standard deviations, with a standard error of 0.775 standard deviations. The local linear regressions are depicted in Panel 1 in Figure 11. Thus, the finding suggests that the null hypothesis of local smoothness in mean QAT 4 scores at the pass rate cutoff cannot be rejected at standard significance levels.

Column 1 in Table 9 presents the RD estimates of the threat or “stick” effect on mean QAT scores for marginal failers among phase-3 program schools. Optimal bandwidth sizes for the local linear regressions, obtained using the I-K method, range between 20–37 percentage 31
points depending on the QAT round. To limit the influence of outliers in outcome values, school observations with QAT 4 pass rates within 15 percentage points of the cutoff with mean QAT score values smaller or larger than the 1<sup>st</sup> and 99<sup>th</sup> percentile values respectively were discarded from the analysis. Figure 11 depicts the local linear regressions of conditional mean QAT scores.

The estimated stick effect on mean scores at the cutoff in QAT 5, the threat QAT, is 0.664 standard deviations and statistically significant. We also find a RD stick effect on mean QAT 6 scores: the estimated effect is 0.561 standard deviations and significant. The inference findings for these QAT rounds are robust to setting the bandwidth to half and twice the optimal bandwidths for the estimations. The estimated RD effect on mean QAT 7 scores, two QATs removed from the threat QAT, drops in magnitude to 0.349 standard deviations and loses significance. The estimated RD effect on mean QAT 8 scores, three QATs removed from the threat QAT, is 0.089 standard deviations; this effect is also not statistically different from zero. The inference results are however not robust to setting the bandwidth to twice the optimal bandwidths for the estimations—when the bandwidths are doubled, the RD effects grow in size and gain significance. However, the observed degressive pattern in the RD effects over QAT rounds remains. Thus, the RD estimates collectively suggest the presence of a strong stick effect on learning for marginal failers. We also find weaker evidence of persistence in the stick effect but this effect appears to dissipate over time once the threat of program exit is no longer immediately present.

The detected RD stick effects on mean QAT scores may be due to QAT 4 failers manipulating the composition of student test takers to increase their chances of attaining the minimum pass rate in QAT 5. We see two opportunities for test-taking pool selection. First, the program school can take advantage of the minimum attendance requirement of 80% for testing to (temporarily) systematically rid itself of its poorest learners before the QAT is administered. Second, the program school can screen in and admit better (potential) learners in anticipation of their stronger performance in the QAT. QAT 4 failers may have more pursued both strategies more intensively than QAT 4 passers. While we cannot directly examine whether the composition of QAT takers have systematically changed over QAT rounds in a way that suggests positive test-taking pool selection, we can indirectly examine this question by studying whether we find any local discontinuities in the conditional expectation of the _number_ of QAT takers at the pass rate cutoff.

32
Column 2 in Table 9 presents the RD estimates of the stick effect on the mean number of QAT takers. Optimal bandwidth sizes for the local linear regressions range between 12–31 percentage points depending on the QAT round. School observations with QAT 4 pass rates within 15 percentage points of the cutoff with test-taker values smaller or larger than the 1<sup>st</sup> and 99<sup>th</sup> percentile values respectively were discarded from the analysis. Figure 12 depicts the local linear regressions of conditional mean QAT takers.

The estimated RD effect on QAT 4 takers is –6.1 students; this effect is not statistically different from zero. Likewise, we do not find evidence of a RD effect on QAT 5 takers: the estimated RD effect is –4.2 students. The inference findings for QATs 4 and 5 are robust to setting the bandwidth to half and twice the optimal bandwidths for the estimations. In QAT 6-8, at the optimal bandwidths for the respective estimations, we also fail to reject the null hypothesis of local smoothness in the mean number of test takers at the cutoff. The inference findings are however not robust to setting the bandwidth to twice the optimal bandwidths for the estimations: the magnitude of the RD effects increases to roughly –15 students and the effects are significant at standard significance levels. Given that there is consistent evidence of local smoothness in mean test takers at the cutoff in QAT 5 (the threat QAT), we discount the sensitivity of the results in the post-threat QAT rounds and read the collective results as suggesting that test-taking pool selection is unlikely to be an important gaming strategy pursued by failers in order to raise mean QAT scores.

# _B. Carrot effect on learning_

The FAS program also awards annual individual cash bonuses to a fixed number of teachers in qualifying program schools. With each annual round of bonuses, PEF has revised the qualifying criteria, largely motivated by the need to limit the number of bonus recipients. The first round of teacher bonuses was offered in January 2007, some 10 months into the academic year for program schools. The program schools that qualified for the bonus were those that achieved a minimum student pass rate of 90% on QAT 2, administered in November 2006. There were a total of 23 schools that qualified for teacher bonuses, which represented 43% of the program schools that took QAT 2. The second round of teacher bonuses was offered in January 2008. The qualifying schools were those that achieved _consecutive_ minimum pass rates of 90% on QAT 3 and QAT 4, administered in March 2007 and November 2007, respectively. There were a total of 33
24 schools that qualified for teacher bonuses in the second round, which represented 12% of the program schools that took QAT 3. The third round of teacher bonuses was offered in January 2009. The qualifying schools were those that achieved the minimum pass rate of 90% in QAT 5, administered in February 2008, as well as obtained a minimum value of 100 in a simple average of the number of students tested in the QAT and the percent mean QAT score across all tested subjects. There were a total of 42 schools that qualified for teacher bonuses, which represented 4% of the program schools that took QAT 5.

Given the small number of schools that qualify for teacher bonuses, the sample for the analysis is expanded to include all program schools that took QAT 5. There were 1,083 schools that took QAT 5. If bonus qualification was solely based on obtaining a pass rate of 90%, 870 schools (80.3%) would have qualified. In contrast, if bonus qualification was solely based on obtaining a minimum score of 100 in the composite measure, 49 schools (4.5%) would have qualified. Applying the two criteria together yields 42 qualifying schools. Consequently, facilitating the application of a RD design to the data, it turns out that the general binding constraint for determining bonus qualification is the minimum score on the composite measure. Figure 13 depicts the independent effects of the two thresholds as well as their combined effect. Thus, the analysis of the carrot effect on learning is conducted on program schools which obtained the minimum pass rate of 90% in QAT 5 and are in the neighborhood of the minimum QAT 5 composite score. Discussions with PEF and a review of program documents indicate that the same thresholds are not used to determine eligibility for any other benefits (or penalties).

Analogous to the threat effect investigation, a sharp RD design is applied to examine the carrot effect for marginal bonus nonqualifiers among program schools who obtained the minimum QAT 5 pass rate of 90%. Given that the composite score is constructed from both learning and test participation measures, the carrot effect may manifest itself in mean QAT scores and/or the number of students that take the QAT (QAT takers). Thus, let<sup>_y_</sup> _s_<sup>denote either</sup> the mean QAT score in the core subjects of Urdu, English, and mathematics or the number of QAT takers in program school _s_ . Let the indicator variable<sup>_d_</sup> _s_<sup>denote treatment assignment,</sup> where one denotes that the school failed to attain the minimum QAT 5 composite score, and zero otherwise. Finally, let _y_ 0 _s_ and _y_ 1 _s_ denote the potential outcomes of school _s_ in the untreated and treated states, respectively. Treatment status is assigned based on the decision rule 34
> _ds_  _zs_  1  _zs_  _c_  , where<sup>_z_</sup> _s_<sup>denotes school</sup><sup>_s_’s QAT 5 composite score which is perfectly</sup> observed, _c_ the known, distinct pass rate cutoff of 100 points, and 1 an indicator function. The conditional probability of treatment as a function of the assignment variable _z_ , Pr  _d s_  1 _zs_  _c_ <sup>,</sup> dives discontinuously from one to zero at the cutoff.

Under the assumptions for sharp RD identification stated before, the carrot effect of the teacher bonus at the cutoff for program schools,  _y_ , is identified by the difference in post-bonus mean outcomes between marginal nonqualifiers and marginal qualifiers. A consistent estimate of the carrot effect at the cutoff is given by  ˆ _y_   ˆ _y_   ˆ _y_  ,<sup>which we estimate via local linear</sup> regressions on either side of the cutoff with kernel _Kh_ and bandwidth _h_ fixed following the same procedures as in the threat effect investigation. In all estimations, sensitivity of the inference results is checked by setting the bandwidth to half and twice the optimal bandwidth. The carrot effect on learning remains identified in the potential presence of mean reversion from the top of the distribution as well as natural learning depreciation as the conditional expectations of these phenomena are assumed to be locally smooth at the composite score cutoff.

We begin by discussing some basic model specification tests. Figure 14 depicts the frequency distribution of program schools over QAT 5 composite scores. Each bin is two points wide and the vertical line represents the bonus qualification cutoff of 100 points. A visual inspection of the frequency distribution shows a nontrivial mass of observations in the neighborhood of the cutoff. The inspection also shows that there is no unusual discontinuity in the number of schools at the cutoff. Table 10 presents summary statistics on the pre-program number of students, number of teachers, maximum and minimum monthly teacher salaries, number of classrooms, gender, location, level, registration status and mean SLQAT score, as well as the sharp RD estimates of local discontinuities in the conditional expectations of these covariates at the cutoff using local linear regressions. Given that program application data are only available for phase-3 and phase-4 program schools, the total sample size for the local smoothness checks for pre-program covariate means decreases from 870 to 658 schools; importantly, only 30 schools qualify for bonuses in the sample. In contrast to the threat effect analysis, we do not find uniform evidence of local smoothness in the conditional means of preprogram covariates at the composite score cutoff: the RD effects for the means of number of teachers, share of schools that is rural, and the share of schools that is coeducational are 35
statistically significant. However, these findings are not robust to setting the bandwidths to half and twice the I-K derived optimal bandwidths for the estimations.

We test if the conditional expectations of mean scores and test participation in QAT 5 (the bonus assignment QAT) exhibit local smoothness at the composite score cutoff. This directly tests a necessary condition for identification under RD. Table 11 shows that the RD estimate of the carrot effect on mean QAT 5 scores at the cutoff is −0.60 standard deviations, with an estimated standard error of 0.60 standard deviations. The local linear regressions are depicted in Panel 1 in Figure 13. Similarly, the RD estimate of the carrot effect on QAT takers at the cutoff is –2.56 students, with an estimated standard error of 3.81 students. The local linear regressions are depicted in Panel 1 in Figure 15. The findings suggest that the null hypothesis of local smoothness in mean learning and test participation levels at the QAT 5 composite score cutoff cannot be rejected at standard significance levels.

Turning now to the results, Table 11 presents the RD estimates of the carrot effect for marginal bonus nonqualifiers among program schools that satisfy the QAT 5 minimum pass rate of 90%. Columns 1 and 2 present RD estimates for the carrot effect on mean QAT scores and QAT takers, respectively. RD effects controlling for pre-program covariates are not estimated given the limited degrees of freedom for the estimations in the cutoff neighborhood. Optimal bandwidth sizes for the local linear regressions, obtained using the I-K method, were roughly 5 points for mean QAT scores and 17 points for QAT takers. Figure 15 depicts the local linear regressions of the conditional mean QAT scores, separately by QAT round. Similarly, Figure 16 depicts the local linear regressions of the conditional mean QAT takers, separately by QAT round.

A priori, we expect no carrot effect in QAT 6, the QAT round just preceding the bonus award, and a carrot effect in QAT 7, the QAT round just following the bonus award, with possible persistence in the effect in QAT 8, the next QAT round. In terms of QAT performance, the estimated effect on mean scores at the cutoff in QAT 6 is −0.87 standard deviations. Although large in size, this estimate is not statistically different from zero. The estimated RD effect on mean scores in QAT 7, the immediate post-bonus QAT, is 0.21 standard deviations and not statistically different from zero. The estimated RD effect on mean scores in QAT 8, the postbonus QAT once-removed, is 0.49 standard deviations and also not statistically different from zero. In terms of QAT participation, the estimated RD effect on QAT 6 takers is –2.6 students; 36
this effect is not statistically different from zero. Likewise, we find no evidence of a RD effect on QAT takers in QAT 7. In QAT 8, we find a significant, negative RD effect on the number of QAT takers—the finding is however not robust to setting the bandwidth to twice the optimal bandwidth for the estimation.<sup>17</sup> The evidence collectively suggests that, despite their size, group-based teacher bonuses do not provide sufficiently strong incentives for marginal nonqualifiers to raise either QAT performance or participation in order to increase their chances for qualifying for bonuses in the next award round. It is plausible that given that PEF has revised the bonus eligibility criteria for every award round (without communicating the revisions well in advance), it may have generated uncertainty on the applicable criteria for the next award round, discouraging the effort of marginal disqualifiers to raise learning and test participation. Independent of this conjecture, it is plausible that gains in learning and test participation by marginal qualifiers may have been masked by gains in the corresponding measures among marginal nonqualifiers. Estimation issues may also be relevant: the inability to detect any effects at the composite score cutoff could be due to inadequate statistical power.

# **8. Conclusion**

Low student learning is a consistent finding in much of the developing world and the search for solutions is an active research and public policy agenda. This paper uses a relatively unique dataset of five semiannual rounds of standardized test data to characterize and explain the shortterm evolution of student learning in primary and secondary schools. The data are collected as part of the quality assurance system for a public-private partnership program which offers public subsidies conditional on minimum learning levels to low-cost private schools in Pakistan. Apart from a large positive distributional shift in learning between the first two test rounds, the learning distributions over test rounds show little progressive movement. School-level panel regressions essentially show that pre-program learning levels and the level of the school are associated with the level and evolution of conditional mean learning. Variance component decomposition shows that between 60–70% of the cross-sectional student-level test score variation is attributable to between-student variation, with no discernible trend in either the level or component shares of

> 17 The effect remains statistically significant when the selected bandwidth is set at half the optimal bandwidth.

37
variation over test rounds. The share of test score variation attributable to between-grade variation is only somewhat smaller than the share attributable to between-school variation.

Schools are ejected from the program if they fail to achieve a minimum pass rate in the test in two consecutive attempts, making the test high stakes. Sharp regression discontinuity (RD) estimates show that the threat of program exit on marginal first-time failers induces large learning gains. The large change in learning between the first two test rounds is likely importantly attributable to this accountability pressure given that a large share of new program entrants failed in the first test round. Schools also qualify for substantial annual teacher bonuses if they de facto achieve a minimum score in a composite measure of student test participation and mean test score. Sharp RD estimates however do not show that the prospect of future teacher bonus rewards induces learning gains for marginal bonus nonqualifiers. Thus, the evidence collectively suggests that, apart from the pressure from below to maintain a minimum level of learning for program participation, program schools do not face any effective incentives to _continuously_ raise learning.

38