metadata: authors: 0: author: Barrera-Osorio, Felipe
1: author: Raju,Dhushyanth
count: Pakistan
docna: 0: docna: Evaluating public per-student subsidies to
            low-cost private schools : regression-discontinuity evidence
            from Pakistan
docty: Policy Research Working Paper
subtopic: Sanitation and Sewerage,Economics of Education,Educational Sciences,Educational Institutions & Facilities,Gender and Development,Education Finance,Effective Schools and Teachers
teratopic: Education
abstracts: cdata!: This study estimates the causal effects
            of a public per-student subsidy program targeted at low-cost
            private schools in Pakistan on student enrollment and
            schooling inputs. Program entry is ultimately conditional on
            achieving a minimum stipulated student pass rate (cutoff) in
            a standardized academic test. This mechanism for treatment
            assignment allows the application of
            regression-discontinuity (RD) methods to estimate program
            impacts at the cutoff. Data on two rounds of entry test
            takers (phase 3 and phase 4) are used. Modeling the entry
            process of phase-4 test takers as a sharp RD design, the
            authors find evidence of large positive impacts on the
            number of students, teachers, classrooms, and blackboards.
            Modeling the entry process of phase-3 test takers as a
            partially-fuzzy RD design given treatment crossovers, they
            do not find evidence of significant program impacts on
            outcomes of interest. The latter finding is likely due to
            weak identification arising from a small jump in the
            probability of treatment at the cutoff.
geo_reg_key: 517198,517212,517191
repnme: repnme: Evaluating public per-student subsidies to
            low-cost private schools : regression-discontinuity evidence
            from Pakistan
display_title: Evaluating public per-student
            subsidies to low-cost private schools :
            regression-discontinuity evidence from Pakistan
listing_relative_url: /research/2011/04/14103503/evaluating-public-per-student-subsidies-low-cost-private-schools-regression-discontinuity-evidence-pakistan
topicv3: Teachers Management m1326119 1523,Education m1327048 470,Private Education m1327271 1490,Gender m1327621 615,Latrines m1505177 3454
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Education
no_of_pages: 62
author_keys: 0: author_key: 237905
model_extractions: WPS5638 Policy Research Working Paper 5638 Evaluating Public Per-Student Subsidies to Low-Cost Private Schools Regression-Discontinuity Evidence from Pakistan _Felipe Barrera-Osorio Dhushyanth Raju_ The World Bank South Asia Region Education Team & Human Development Network Education Team April 2011
Policy Research Working Paper 5639

# **Abstract**

This study estimates the causal effects of a public perstudent subsidy program targeted at low-cost private schools in Pakistan on student enrollment and schooling inputs. Program entry is ultimately conditional on achieving a minimum stipulated student pass rate (cutoff) in a standardized academic test. This mechanism for treatment assignment allows the application of regressiondiscontinuity (RD) methods to estimate program impacts at the cutoff. Data on two rounds of entry test takers (phase 3 and phase 4) are used. Modeling the entry process of phase-4 test takers as a sharp RD design, the authors find evidence of large positive impacts on the number of students, teachers, classrooms, and blackboards. Modeling the entry process of phase-3 test takers as a partially-fuzzy RD design given treatment crossovers, they do not find evidence of significant program impacts on outcomes of interest. The latter finding is likely due to weak identification arising from a small jump in the probability of treatment at the cutoff.

This paper is a product of the Education team, South Asia Region; and the Education Team, Human Development Network. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http:// econ.worldbank.org. The authors may be contacted at fbarrera@worldbank.org and draju2@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Evaluating public per-student subsidies to low-cost private schools: Regression-discontinuity evidence from Pakistan

## Felipe Barrera-Osorio Dhushyanth Raju

_JEL classification codes_ : I21; I28; O10 _Keywords_ : education; Pakistan; regression-discontinuity design; subsidies; private schools Contact information: Barrera-Osorio: World Bank, <u>fbarrera@worldbank.org;</u> Raju: World Bank, <u>draju2@worldbank.org. We thank the staff of the Punjab Education Foundation, in particular Usman Tariq Bhatty,</u> Mian Kashif Ijaz, Shahid Hafiz Kardar, Allah Bakhsh Malik, Haroon Raheem, and Huma Rizvi, for extensive discussions on program design and implementation and assistance with the program administrative data; Nabiha Asghar, Hummayun Javed, Kashif Mahmood, and Sehar Sarah Sikander Shah for assistance with the follow-up data collection; Corinne Saiens for assistance with the National Education Census data; Amit Dar, Michelle Riboud, Sofia Shakil, and the Government of Punjab School Education Department/Project Management and Implementation Unit for their encouragement and support of the research project; and Salman Asim, Elizabeth King, Leigh Linden, and Quynh Nguyen for helpful comments and suggestions. Financial support for the follow-up data collection from the World Bank’s Education Program Development Fund and Bank-Netherlands Partnership Program Trust Funds is gratefully acknowledged. The findings, interpretations, and conclusions expressed herein are our own and do not necessarily represent the views of the World Bank, its Board of Directors, or any of its member countries. All remaining errors are our own.
# **1. Introduction**

School participation in Pakistan is generally low in absolute terms, relative to other countries in its region, and relative to developing countries at its level of per capita income. Given the present trend, Pakistan is likely to fall significantly short of the United Nations’ Millennium Development Goal of universal primary education by 2015. Thus, measures that produce sharp and sustained gains in participation are viewed as pressing by development policymakers and practitioners in the country.

In this paper, we estimate the causal effects of a recently-instituted public subsidy program to low-cost private schools in the province of Punjab, Pakistan on school enrollment and inputs. Coined the Foundation Assisted Schools (FAS) program and administered by the Punjab Education Foundation (PEF), a semi-autonomous intermediary organization, the rationale behind the program is to leverage, essentially via public financing, the growing low-cost private school system in Punjab to increase equitable access to schooling more efficiently than can be achieved through the province’s public school system.

Given the contracting modality, the program falls under the rubric of public-private partnerships (PPPs) in education, a set of interventions which is increasingly perceived in international policy circles as a promising mechanism for attaining key education goals (World Bank 2009) _._ In addition, opportunities for introducing PPP programs of medium to large scale are emerging in several developing countries (e.g., India, Kenya, and Nigeria) as the private education sector matures and becomes an important player in service delivery.

The FAS program was initiated on a pilot basis in November 2005, and rapidly expanded in phases. As of September 2008, the program had completed four phases of expansion and 2
covered 1,082 low-cost private schools (primary, middle, and secondary levels<sup>1</sup> ) with roughly 474,000 students across 18 out of 35 districts in Punjab. In the districts with the highest concentration of program schools, we estimate that the program covers approximately one-fifth of all private schools. The program offers a monthly per-student cash subsidy strictly tied to, among other things, free schooling for all enrolled children (thus, pricing program schools in line with public schools which are officially free) and a minimum student pass rate in a speciallydesigned standardized academic test administered semi-annually by PEF. Program schools have free rein on how to spend the subsidy amount.<sup>2</sup> To the best of our knowledge, credible evidence on the impacts of public subsidies to private schools is limited. The study by Kim et al. (1999) is particularly pertinent as they evaluate a public subsidy program in Pakistan which shares some of the design elements of the FAS program. They study the impact of a program that offered a low temporary per-girl student subsidy conditional on free girls’ schooling to establish and operate private primary schools in a randomly-selected subset of poor urban neighborhoods lacking public girls’ primary schools in the province of Balochistan. Using a difference-in-differences approach, they find that the program substantially increased girls’ as well as boys’ school participation in treatment neighborhoods, and that these increases were obtained at lower costs than would have been possible through the public school system. For other well-identified evaluation studies of subsidy programs (as well as of other types of PPP interventions), see the recent review by World Bank (2009).

> 1 Primary schools are composed of grades 1–5. Middle schools are composed of grades 1–8 or 6–8. Secondary schools are composed of grades 1–10, 6–10, or 9–10.

> 2 In contrast, the standard subsidy program is designed to directly finance educational inputs (Gauri and Vawda 2003).

3
Consistent with the objectives of the FAS program, we ask two questions in this study. (1) What is the causal effect of the program on the number of students in program schools? (2) What is the causal effect of the program on inputs, namely the number of teachers, classrooms, blackboards, and toilets, and on student-teacher and student-classroom ratios in program schools? To answer these questions, using school-level baseline and follow-up data on school characteristics and outcomes obtained from program administrative records and telephone interviews of school administrators, we fit appropriate regression-discontinuity (RD) designs to the treatment assignment mechanism in order to obtain reliable nonexperimental estimates of program impacts.

In the last two entry phases preceding this study, phases 3 and 4, a standardized academic test, called the Short Listing Quality Assurance Test (SLQAT), was administered by PEF as the final step in the program entry screening process. In order for schools to enter the program in these two phases, they had to apply to the program, pass a qualitative physical inspection, and then pass the SLQAT. If the school achieves the stipulated minimum student pass rate (the cutoff) in the SLQAT, then the school becomes eligible for the program, and not otherwise. Furthermore, in practice, virtually all schools that become eligible elect to participate in the program.

At the time of the follow-up data collection in October 2008, as phase 4 was the last entry phase, schools that took the phase-4 SLQAT were either untreated or treated based on their SLQAT pass rate relative to the cutoff (i.e., the probability of treatment jumps from zero to one at the cutoff). This structure allows us to apply a sharp RD design to these data. On the other hand, schools that took the phase-3 SLQAT and failed found they had another opportunity to seek entry when phase 4 was announced—some phase-3 SLQAT ―failers‖ reapplied to phase 4, 4
recleared the physical inspection, retook the SLQAT, and passed it. We find that the probability of treatment for phase-3 SLQAT takers also jumps at the cutoff but by less than one. This structure allows us to apply a special case of a fuzzy RD design which accounts for what can be viewed as equivalent to noncompliance by initially-untreated schools.

Given these designs, the effects of the program are identifiable and estimable. In the case of the sharp RD design, under some mild regularity conditions, the average causal effect of the treatment on the treated at the cutoff is identified. In the case of our fuzzy RD design, under the same regularity conditions, the average causal effect of the treatment on the _untreated_ at the cutoff is identified. Under both designs, the respective treatment effects at the cutoff are estimated nonparametrically using local linear regressions.

Our findings on the impacts of the FAS program differ by phase. For phase-3 SLQAT takers, applying a partially-fuzzy RD design, we find no evidence of program impacts at the cutoff on our outcomes of interest. We posit that this is likely due to weak identification arising from a small jump in the probability of treatment at the cutoff. In contrast, for phase-4 SLQAT takers, applying a sharp RD design, we find robust evidence of positive program impacts at the cutoff on the number of students, teachers, classrooms, and blackboards. The program impacts were sizeable both in absolute terms as well as relative to the baseline means for these outcomes for phase-4 SLQAT takers near the cutoff: our most conservative estimates suggest that the program expanded schools by roughly 85 students, and 3–4 teachers, classrooms, and blackboards, which translate into relative increases of 27–47%, depending on the outcome of interest. These impact estimates are particularly impressive when we consider that phase-4 program schools were exposed to only about ten months of treatment before the follow-up data were collected. Our cost-effectiveness analysis suggests that the FAS program is among the 5
cheapest interventions in developing countries for generating enrollment gains. Notwithstanding, the estimated impact on enrollment needs to be interpreted cautiously as, given our data, we are unable to establish the extent to which the enrollment gains translate into school participation gains given that, among other things, the elimination of school fees in program schools is likely to also attract students from non-program schools in the vicinity of program schools.

The remainder of the paper is organized as follows. Section 2 describes the education context in Pakistan at the time the FAS program was introduced. Section 3 describes the program in detail. Section 4 lays out our identification and estimation strategies. Section 5 describes the data and provides some summary statistics for SLQAT takers. Section 6 presents our impact and cost-effectiveness findings. Section 7 summarizes and interprets our main findings, as well as provides some concluding comments. These comments comprise of a discussion of external validity and two specific potential threats to internal validity: treatment spillovers to nonprogram low-cost private schools within the sphere of influence of program schools, and behavioral changes in non-program schools in anticipation of future program entry.

# **2. The education context**

The FAS program was conceived and introduced into an education landscape defined by three features. One: equitable access to schooling is an acute, persistent challenge. Two: the public sector, which remains the dominant provider of education, suffers from chronic weaknesses which impair its ability to effectively address this challenge. Three: in the wake of this public sector failure, the private sector has aggressively stepped in in recent years, growing dramatically in size and reach, and now constitutes a major policy opportunity for addressing this challenge.

6
The education situation of Pakistan is generally poor in absolute terms, relative to other countries in its region, and relative to developing countries at its level of per capita income (see, e.g., United Nations Educational, Scientific and Cultural Organization 2007 for a recent comparative picture). Given the present trend, Pakistan is unlikely to meet the United Nations’ Millennium Development Goal of universal primary education by 2015. The education situation of Punjab—Pakistan’s most prosperous and populous province and the site for the FAS program—is by and large comparable to the rest of the country. As Panel A in Table 1 shows, estimates using household sample survey data from 2004/05, which is just before the FAS program was initiated, show that the participation rate in formal school (grade 1+) of children ages 6–15 years was 65.7%, with a significantly lower share for children from the poorest (bottom expenditure quintile) households.

The public school system has been hampered in its ability to improve education outcomes due to, in large part, the lack of effective accountability and incentive systems which promote the legitimate and efficient use of allocated resources (Social Policy and Development Center 2003).<sup>3</sup> While the public school system has struggled to enroll children and educate them, the private school system has grown dramatically as reflected by the number of institutions and share of enrollment. In addition, responding to the broad demand for greater access and better quality, the system has evolved in character, becoming less elite and more egalitarian, increasingly

> 3 Recent evidence on relative teacher performance in private and public schools in Punjab suggests this. Andrabi et al. (2008a) find that, while public school teachers are on average significantly better paid and have higher levels of education, training, and experience than their counterparts in private schools, the teacher absenteeism rate is 15% in public schools compared to 8% in private schools. They also find that, whereas private school teacher salaries are increasing in teacher competency (as measured by teacher test scores) and decreasing in teacher absenteeism, public school teacher salaries do not exhibit similar correlation patterns and instead appear to be largely determined by teacher credentials (education, training, and experience). Looking at child education outcomes, Andrabi et al. find that private school teacher salaries are increasing in student test scores whereas public school teacher salaries are not. The responsiveness of private school teacher salaries to these key behaviors and outcomes suggests the influence of market discipline which the public school system is not subject to, at least directly.

7
reaching low-income and rural households.<sup>4</sup> Using private school census data from 2000, Andrabi et al. (2008b) find that there was an exponential increase in the number of private schools over the 1990s, with over 50% of existing private schools established in or after 1996. Furthermore, they find that while existing schools set up before 1990 were predominantly in urban areas, the distribution since then has become increasingly rural.

In line with this increase in institutions, the share of enrollment in private school has also increased. As Panel B in Table 1 shows, in 2004/05, 15.8% and 18.5% of children ages 6–15 years were enrolled in private schools in all-Pakistan and Punjab, respectively; these shares represent significant increases from 1998/99. The increases were particularly dramatic in rural areas and for households in the poorest expenditure quintile. Similar evidence is provided by Andrabi et al. (2008b) who find that the growth rate in private school enrollment over the 1990s was highest among low-income households nationally, and among middle-income households in rural areas, whereas the growth rate in public school enrollment over the same period was negative in both urban and rural areas and across the household income distribution. They also find that fees in private schools are generally low: median annual fees per student in 2000 in Pakistan were 960 (US$23.4<sup>5</sup> ) and 751 rupees (US$18.3) in urban and rural areas, respectively, and account for a small percentage of mean annual household expenditure. The corresponding statistics for Punjab were lower, at 828 rupees (US$20.2) and 600 rupees (US$14.6).

> 4 It is conceivable that much of the growth and metamorphosis in the private school system is in direct response to the rigidities and shortcomings in the public school system.

> 5 The exchange rate in 2000 was 41 rupees per US dollar.

8
# **3. The Foundation Assisted Schools program**<sup>**6**</sup>

_Institutional background_ : The FAS program is administered by PEF, a publicly-funded semiautonomous statutory organization established in 1991 which serves as the main institutional mechanism for PPPs in education in Punjab. The organization’s primary aims are to enable socioeconomically-disadvantaged households to access private education and to raise the quality of education in low-cost private schools. To these ends, it employs an array of instruments such as start-up and operational subsidies, vouchers to households in poor localities, school management and teacher training, and specialist teacher services to promote private education. The FAS program is PEF’s largest program. In fiscal year 2007–08, PEF spent 1.1 billion rupees (US$12.9 million<sup>7</sup> ) on FAS program benefits.<sup>8</sup> This amount accounted for 61% of total expenditures by the organization in that year.

_Program coverage and timeline_ : The FAS program was initiated in November 2005 on a pilot basis in 54 schools in five districts in Punjab. Since then, PEF has expanded the program in phases to cover additional districts as well as more schools within program districts. (See Table 2 for a timeline of the program highlighting the start-dates of each of the phases as well as the dates of other key events.) As of September 2008, the program has completed four entry phases (the pilot phase represents phase 1), and covers 1,082 private primary, middle, and secondary schools in 18 out of the 35 districts in Punjab.<sup>9</sup> Of these, 945 schools (87%) are located in just seven districts (see Table 3 for a district- and phase-wise disaggregation of the number of program schools as well as Figure 1 for the location of program districts in Punjab). This number

> 6 The information on the program presented in the paper reflects program design and administration until end 2008. 7 In this and following sections, the exchange rate used for the conversions is 85 rupees per US dollar (effective March 2011). When the program was introduced in 2005, the exchange rate was roughly 60 rupees per US dollar; the rupee has steadily weakened since then.

> 8 FAS program administrative costs are unavailable as PEF does not disaggregate administrative costs by program. We however know that total administrative costs were less than 1% of total expenditures in fiscal year 2007–08, suggesting that FAS program administrative costs are negligible relative to current FAS program benefit outlays.

> 9 The FAS program also currently covers three higher secondary schools (schools with grades 11–12).

9
of program schools represents a significant share of all private schools in these seven districts: using the 2005 National Education Census (NEC), a census of schools in Pakistan, we estimate that the program covers 21% of private schools in these districts (see Table 4).<sup>10</sup> _Program location_ : The program was initially designed to be targeted at districts ranked lowest in terms of adult literacy rates based on 2003/04 Multiple Indicators Cluster Survey data (Government of Punjab 2004). However, in phases 1 and 2, this targeting decision was not applied as PEF was limited in its institutional and logistical capacity at that time to effectively administer the program in poorly-ranked districts that had physical environmental challenges and limited transportation and accommodation options (Malik 2007). In contrast, in phases 3 and 4, the targeting decision was effectively applied. Consequently, as Table 3 shows, 51% and 89% of program schools are located in districts ranked among the bottom-quarter and bottom-half in terms of adult literacy rates, respectively.

_Program school characteristics_ : Table 5 presents the distribution of program schools by selected characteristics measured in September 2008. Aggregating the phases together (Column 5), the mean school size is 351 students. The majority of schools are middle level (59%), coeducational (83%), registered with local government authorities (87%), and rural (55%). Disaggregating by phase (Columns 1–4), the distribution of program schools by these characteristics is roughly comparable except for the level of the school: in contrast to the pattern in the aggregate sample, program schools that entered in phases 1 and 2 were mainly secondary schools (65–73%). School size also appears to be monotonically decreasing with phase; the mean size of program schools which entered in phase 1 is 561 students, whereas that of program schools which entered

> 10 NEC’s coverage of private schools might be incomplete. According to the survey documentation, private schools were identified in the field by interviewers with the assistance of local officials. It is conceivable that private schools that are, for example, unregistered or very small, or operate in obscure locations are more likely to be missed by interviewers. Thus, the estimate of 21% can be considered as an upper-end estimate of FAS program coverage of private schools.

10
in phase 4 is 242 students. While there may be multiple (competing) explanations for this pattern, length of exposure to the program is among them.

_Program benefits_ : The main program benefit is an enrollment-related subsidy: the school receives a monthly per-student cash subsidy amount of 300 rupees (US$3.5) up to a maximum of 750 students (i.e., the total amount is capped at 225,000 rupees or US$2,647).<sup>11</sup> Given a mean school size of 215 students in phase-3 and phase-4 program schools at the time of application to the program, the mean monthly subsidy payment is roughly 64,500 rupees (US$759) at program entry. Enrollment information for determining the subsidy amounts is submitted by program schools to PEF on a monthly basis using standardized reporting forms; if enrollment has increased by 50 students or more over one month, PEF visits the school to verify the information before raising the subsidy amount. PEF indicates that, when schools report large changes in enrollment, these reports tend to be at the start of the academic year in April.

PEF reports that the subsidy level was set low for two reasons. First, it confines the attractiveness of the program to low-cost private schools. Second, it raises the political palatability of the program as the per-student subsidy amount is less than half of the estimated per-student expenditure in the public school system at the time the program was introduced.<sup>12</sup> The subsidy benefit is paid for all twelve months of the year. To facilitate timely and regular payments, starting in August 2007, the benefit amounts have been transferred electronically to the bank accounts of program schools.

> 11 The program also offers two cash bonus benefits. The first is a teacher bonus for a high level of school test performance: once every academic year, a maximum of five teachers in each program school where at least 90% of students in tested classes obtain a score of 40% or higher in the QAT receive an award of 10,000 rupees (US$118) each. The second is a competitive school bonus for top school test performance: once every academic year, the program school in each of the seven main program districts which has the highest share of students with a score of 40% or higher in the QAT is awarded 50,000 rupees (US$588).

> 12 The exact subsidy amount was heavily guided by a survey conducted by PEF in 2005 in selected districts which showed that the vast majority of private schools that operate in rural areas and disadvantaged urban neighborhoods charge between 50–400 rupees per month (US$0.6–4.7) in fees. Based on this information, the subsidy amount was set at the upper-segment of this price range (Malik 2007).

11
_Initial benefit eligibility rules_ : School entry into the program follows a three-step process. In step 1, schools apply to the program when PEF issues a call for applications in newspapers. Application eligibility is restricted to existing private primary, middle, and secondary schools with a minimum enrollment of 100 students from the districts listed in the call. Except in a few cases, only schools that submit properly filled-in applications by the announced deadline are considered for step 2.

In step 2, PEF inspection teams visit schools unannounced to verify the data provided in the applications as well as assess the local reputation of the school and the quality of the physical infrastructure and schooling environment. PEF provides a points and weighting scheme to its inspection teams—this scheme primarily helps ensure that the same school attributes are considered across schools and by different inspection teams. However, the screening exercise is largely subjective: whether a school qualifies for step 3 depends principally on qualitative impressions gathered by the inspection team on a given set of indicators.

In step 3, which was introduced starting with phase 3, all students present in selected grades in the school on the day of the physical inspection screening are offered a 55-minute, written, curriculum-based test called the Short Listing Quality Assurance Test (SLQAT).<sup>13</sup> The SLQAT is a pared-down version of the Quality Assurance Test (QAT) which is used to determine continuing benefit eligibility once schools enter the program. The SLQAT tests knowledge and comprehension in three subjects: English, Urdu, and mathematics. It is designed by the testing department of PEF, the Academic Development Unit (ADU). Each school is tested on its own premises. Depending on the level of the school, two to three grades among grades 3, 4, 6, 7, and 9 are tested. Which grades are tested in a given school are not disclosed in advance and are randomly selected by ADU. Furthermore, multiple test papers are prepared for each

> 13 For schools that entered the program in phases 1 and 2, step 2 was the final entry step.

12
grade; ADU randomly selects which specific test papers are offered in a given school. The SLQAT is transported in sealed envelopes and opened only in the presence of the school administrator and teachers. At that time, both the PEF inspection team and the school learn which grades are selected for the SLQAT. Test invigilation is coordinated by the ADU member assigned to the PEF inspection team. The completed tests are transported back to PEF headquarters and graded by ADU. The school passes the SLQAT if at least 67% of tested students score 33% or higher.

Figure 2 presents the numbers of schools which passed each step of the program entry process. Out of the 1,070 and 1,430 schools that had submitted properly filled-in applications and were inspected in phases 3 and 4, respectively, 799 (75%) and 872 (61%) schools were offered the SLQAT. Of the schools that were offered the SLQAT, 514 (64%) and 431 (49%) schools achieved the minimum pass rate and became eligible for the program.<sup>14</sup> In terms of program take-up, PEF reports that 482 (94%) and 425 (98%) of the schools that passed the SLQAT in phases 3 and 4 respectively signed the formal program participation agreement.

_Continuing benefit eligibility rules_ : The program participation agreement stipulates several conditions for maintaining benefit eligibility. The conditions that are stringently applied by PEF are (1) schooling is offered to students without charging them any fees (and displaying the freeschooling status prominently on a PEF-issued signboard outside the school gate) and (2) participation of the program school in the QAT and that at least 67% of the tested students score 40% or higher on the QAT. A one-time violation of these conditions typically results in a warning and the capping of enrollment figures for the subsidy payment until the next QAT round. A second violation results in the permanent disqualification of the school with immediate

> 14 Our own tallies based on the SLQAT data deviate slightly from the above numbers: we find that 796 and 856 schools took the SLQAT and, out of these, 511 and 432 schools achieved the minimum pass rate in phases 3 and 4, respectively.

13
effect. With the QAT-related condition, the school is permanently disqualified if it fails to achieve the minimum pass rate in the QAT in two consecutive attempts.

The QAT is a 65-minute, written, curriculum-based test designed by ADU, and administered twice a year in October–November (in the first term) and February–March (in the second term). It tests knowledge, comprehension, application, analysis, and synthesis. The subject areas covered in the tests are English, Urdu, mathematics, and science (general science in grades 1–8 and biology, chemistry, and physics separately in grades 9–10). The general test administration procedures previously delineated with respect to the SLQAT are also followed with the QAT.<sup>15</sup> By the time of the follow-up data collection, PEF had conducted five QATs (see Table 2 for the dates). Phase-3 program schools had been subject to two QATs and phase-4 program schools one.

There are also other conditions for maintaining benefit eligibility. These include (1) registering the school with the District Registration Authority within one year of joining the program; (2) conducting only one class in a classroom in any period; (3) maintaining or upgrading the quality of the school’s physical infrastructure (e.g., adequate classroom space, properly-constructed rooms and buildings, sufficient ventilation, and sufficient artificial and natural light); (4) acquiring and maintaining adequate furniture and teaching tools (e.g., benches, desks, and blackboards); (5) providing monthly reports to PEF on enrollment counts; (6) keeping student-teacher and student-classroom ratios below 35:1; (7) keeping enrollment above 100 students; and (8) not holding after-hours classes or tutoring services at the school. These additional conditions are applied more leniently; typically, when PEF detects a violation among this subset of conditions, schools are provided with a warning and a grace period within which to

> 15 One important difference is that, unlike with the SLQAT, the school receives formal advance notices of the date of the QAT, and at least 80% of its students are expected to be in school on the day of the test.

14
comply. As of the follow-up data collection, no program schools have been disqualified for repeated violations of these conditions.

Table 6 presents the number of program schools that were disqualified, disaggregated by phase. The data are up until the time of the follow-up data collection. The number of schools that have been disqualified for any reason is negligible: only 28 out of the 1,111 program entrants (2.5%) have exited the program over time. In addition, in most cases, the reason for disqualification is two consecutive failures to achieve the minimum pass rate on the QAT. Importantly, for our analysis, only three schools that entered the program in phases 3 and 4 have exited the program for any reason.<sup>16</sup> Thus, program dropout has not been an issue with phase-3 and phase-4 program schools.

_Direct impact channels_ : The structure of the program can be expected to have a positive effect on enrollment via several direct channels. For example, setting the monthly subsidy as a linear function of the number of children enrolled incentivizes program schools to draw in additional students. Tying the receipt of program benefits to the elimination of school fees—which puts the program school’s price at an advantage over other local non-program competitor private schools and at par with public schools—is likely to raise the attractiveness of program schools, particularly among households for which school fees serves as a major constraint to sending their children to private school.<sup>17</sup> The structure of the program can also be expected to directly affect investments in the quantity and quality of school inputs and resources. For example, increases in enrollment

> 16 PEF reports that the three schools were problematical cases from the outset; the schools were ejected from the program for general noncompliance and nonperformance.

> 17 An increase in enrollment in program schools may not translate into a commensurate increase in participation among children in areas where program schools operate. The lower price of schooling in program schools is likely to induce both displacement and diversion effects: some share of the new enrollment in the program school will likely come from students already enrolled in other schools or children that where initially _considering_ enrolling elsewhere.

15
induced by the per-student subsidy have to be met by increases in the number of classrooms and teachers if the stipulated maximum student-teacher and student-classroom ratio conditions are to be complied with. The maximum ratios encourage program entrants with preexisting ratios in excess of the maximums to invest in additional classrooms and teachers. The physical infrastructure and learning environment quality conditions encourage program schools to ensure the proper design and construction of infrastructural expansions, and to invest in teaching tools (e.g., blackboards) and basic facilities (e.g., toilets), in step with enrollment growth. These inputrelated conditions encourage schools to schedule investments in school inputs and resources to either lead or accompany enrollment increases. However, given that these conditions are not stringently applied, it suggests that PEF might tolerate investments in school inputs and resources that lag enrollment increases, although reportedly not by too long.

# **4. Empirical strategy**

# _Identification_

> Following the exposition in van der Klaauw (2008) and Todd (2007), let<sup>_y_</sup> _i_<sup>denote the outcome</sup> of interest (e.g., enrollment) in school _i_ , and let the indicator variable<sup>_d_</sup> _i_  1,0 denote treatment

assignment, where one denotes that the school is covered by the FAS program (treated), and zero if not (untreated). In addition, let _y_ 0 _i_ and _y_ 1 _i_ denote the potential outcomes of school _i_ in the untreated and treated states, respectively. The actual outcome observed for school _i_ is given by

> where  _i_ denotes the treatment effect for school _i_ .

16
Unless treatment is randomly assigned, simply comparing mean outcomes of treated schools and untreated schools, _E_  _yi_ 1 | _di_  1   _E_  _yi_ 0 | _di_  0  , would not yield an unbiased estimate of the average causal effect of the treatment: _E_  _i_  . In general,

The first term on the right-hand side of (2) is the average treatment effect on the treated (ATT) and the second term (in braces) is the selection bias arising from differences in potential average untreated outcomes between treated and untreated schools. Selection is clearly plausible in our case. For example, if, ex ante, program schools have higher levels and better quality inputs and resources relative to non-program schools, then the selection bias term will be positive and generate an upward bias in our estimate of the ATT.

The potential selection bias problem can however be overcome by using the institutional feature that program eligibility is ultimately determined by the student pass rate obtained by the school in the SLQAT relative to the known pass rate cutoff of 67%. Given that virtually all schools that become eligible to participate in the program also choose to participate in the program, in practice, the cutoff determines actual program participation. Thus, program participation status is assigned based on the decision rule where<sup>_z_</sup> _i_<sup>denotes school</sup><sup>_i_’s pass rate which is perfectly observed (</sup><sup>_z_will be more generally</sup> referred to as the assignment variable), _c_ the known distinct cutoff pass rate, and 1  an indicator function. This design for treatment assignment based on a completely deterministic function is referred to as a sharp regression-discontinuity (RD) design.

17
The SLQAT pass rate _z_ might be correlated with our outcomes of interest. Thus, the treatment assignment mechanism is clearly not random and comparing schools that receive the treatment ( _di_ <sup>_z_</sup> _i_  1) to schools that do not ( _di_ <sup>_z_</sup> _i_  0 ) will yield a biased estimate of the ATT. If, however, we consider schools with pass rates near the cutoff to be comparable, then the treatment assignment mechanism in the neighborhood of the cutoff can be viewed as if it was ―almost random‖.

More formally, let _e_  0 denote an arbitrarily small number. Comparing the outcomes of schools with pass rates just below the cutoff (marginal failers) with the outcomes of schools with pass rates at or just above the cutoff (marginal passers), yields

> _E_  _y_ 1 _i_ | _zi_  _c_  _e_   _E_  _y_ 0 _i_ | _zi_  _c_  _e_   _E_  _i_ | _zi_  _c_  _e_   _E_  _y_ 0 _i_ | _zi_  _c_  _e_  _E_  _y_ 0 _i_ | _zi_  _c_  _e_  .(4)

> Under the assumptions that (1) the limit lim _E_  _y_ 0 _i_ | _zi_  _c_  _e_  is well defined; (2) _E_  _y_ 0 _i_ | _zi_  _c_  _e_  0 is continuous in the assignment variable _z_ at the cutoff (i.e., the conditional expectations of the outcome variable exhibits local smoothness at the cutoff in the absence of the treatment); and (3) the density of the assignment variable _z_ is positive in the neighborhood of the cutoff, the difference in the mean outcomes between marginal passers and marginal failers identifies

This object is interpreted as the ATT at the cutoff (Hahn et al. 2001; Todd 2007).

A sharp RD design neatly fits the phase-4 SLQAT taker data, as the school’s pass (eligibility) versus fail (ineligibility) status remains fixed given that phase 4 was the last entry phase before the follow-up data collection. A variant of the above framework will however be required for phase-3 SLQAT takers given that some phase-3 failers reapplied to the program in phase 4 and passed the SLQAT at that time. As a result, our set of phase-3 SLQAT takers can be 18
divided into three distinct subsets: (1) phase-3 failers who did not enter the program any time later; (2) phase-3 failers who entered the program later in phase 4 (crossover schools); and (3) phase-3 passers who entered the program at that time (as noted before, the treatment take-up rate by passers was virtually 100%, and there have been no dropouts to date). While the pass rate cutoff rule strictly determines program eligibility status for phase-3 SLQAT takers, the processes and factors behind the phase-4 application decision of phase-3 failers are unobservable to us and likely to be nonrandom. Given this, the eventual participation status of this group of schools is also considered to be driven partly by selection on unobservables.

In our data, we find that the probability of program participation as a function of the pass rate in the phase-3 SLQAT, _E_  _di_ | _zi_   Pr  _di_  |1 _zi_  , consists of a positive but smaller jump than one at the cutoff. The probability of program participation will still be discontinuous in the assignment variable _z_ at the cutoff. This feature implies that the phase-3 SLQAT taker data are more appropriately fitted with a fuzzy RD design. Under this design, apart from the identifying assumptions for the sharp design, if we assume that treatment<sup>_d_</sup> _i_<sup>is nondecreasing in the</sup> assignment variable _z_ at the cutoff (local monotonicity) and that potential outcomes are independent of treatment conditional on the assignment variable _z_ at the cutoff (local conditional independence), then the average causal effect of the treatment is identified by

where the denominator of (6) is nonzero given the discontinuity in the probability of treatment at the cutoff (Hahn et al. 2001; Todd 2007).

This treatment parameter represents the RD analog to the local average treatment effect (LATE) (Imbens and Angrist 1994; Angrist et al. 1996). It can be interpreted as the average 19
treatment effect for schools in the neighborhood of the cutoff who are induced to take up treatment when their pass rates are at or above the cutoff  _di_ <sup>_z_</sup> _i_  1  but not so when they are

> below  _di_ <sup>_z_</sup> _i_  0  .

The general fuzzy design setup above allows for partial (or imperfect) compliance for both treatment-eligible and treatment-ineligible groups; that is, it allows for both crossovers (treatment-ineligibles taking up treatment) and no-shows (treatment-eligibles not taking up treatment) (Bloom 1984). In our case, the problem is essentially one of one-way partial compliance: we have the equivalent of crossovers over time from treatment-ineligible to treatment-eligible status, with all treatment-eligible schools choosing to take up treatment. Hence, the probability of program participation for phase-3 SLQAT passers is one, while the probability for failers lies between zero and one.

Battistin and Rettore (2008) describe the other one-way partial compliance case, where treatment-ineligible agents are not exposed to treatment and treatment-eligible agents self-select into treatment. They formally show that the LATE at the cutoff can be identified simply under the conditions required for identifying the ATT at the cutoff under the sharp RD design, thus labeling their framework as a ―partially‖ fuzzy RD design. Their framework straightforwardly applies to our case of one-way partial compliance as well. However, whereas in their specific setup, they identify the ATT at the cutoff ( lim _E_  _i_ | _zi_  _c_  _e_  ), by applying the smoothness _e_  0 condition to obtain the counterfactual average outcome for the untreated ( lim _E_  _yi_ 1 | _zi_  _c_  _e_  ), _e_  0 we instead identify the average treatment effect on the _untreated_ ( lim _E_  _i_ | _zi_  _c_  _e_  ), by _e_  0 applying the smoothness condition to obtain the counterfactual average outcome for the treated (

lim _e_  0 _E_  _yi_ 0 | _zi_  _c_  _e_  ) (Duflo et al. 2007).

20
# _Estimation_

_Choice of estimator_ : Given that we are interested in estimating the treatment effects at a single point using observations in its neighborhood, one suitable approach is local smoothing using nonparametric regression. Following Hahn et al. (2001), we opt for local linear regression (a local polynomial of order one). In the case of the sharp RD design, which applies to our phase-4 SLQAT takers, the application of local linear estimation entails individually estimating the ˆ  conditional expectations of the outcome _y_ at the cutoff from below (denoted by  _y_ ) and above ˆ  the cutoff (denoted by  _y_ ) and then subtracting the two estimates. Under the sharp RD design, a consistent estimate of the ATT at the cutoff is given by In the case of the fuzzy RD design, the application of local linear regression entails estimating the ratio of two differences. Given our particular problem related to phase-3 SLQAT takers, we individually estimate the conditional expectations of the outcome _y_ at the cutoff _c_ from above and below (  ˆ _y_  and  ˆ _y_  , respectively), and the conditional expectation of treatment _d_ from below only (denoted by  ˆ _d_<sup></sup> ). Note that the conditional expectation of treatment _d_ from above is one by definition. Under our particular partially-fuzzy RD design, a consistent estimate of the LATE at the cutoff is given by 21
To perform statistical inference, empirical standard errors for (7) and (8) are obtained using the standard i.i.d. nonparametric bootstrap with paired sampling  _y_ , _i_<sup>_z_</sup> _i_  , resampling from the original data 500 times.

_Choice of kernel and bandwidth_ : Implementation of local linear estimation requires the specification of the kernel _k_ , the weighting function, and bandwidth _h_  0 , the window width in which the kernel function is applied. We opt for the triangular kernel given that it is boundary optimal and thus well suited to RD problems (Cheng et al. 1997).<sup>18</sup> The choice of bandwidth is a relatively more important decision given the trade-off between estimation bias and variance. Consensus in the literature on valid and robust methods for bandwidth selection for RD problems is lacking.<sup>19</sup> Given this, we discount the specific method used to select the optimal bandwidth and instead emphasize checking the robustness of our findings to reasonable deviations in bandwidth from the optimal bandwidth choice.

To select the optimal bandwidth, we rely on the default bandwidth choice in Nichols’ (2007) `rd` program for `Stata` which assigns positive weight to at least 30 observations on each side of the cutoff, and applies the same bandwidth for the local linear estimators of the conditional mean outcomes (and probabilities of treatment) above and below the cutoff (i.e.,   _h_  _h_  _h_ ). In our case, the default choice translates to a bandwidth of three or four percentage points for our RD impact estimations. We believe that this is a conservative choice, and

> 18 As Imbens and Lemieux (2008) note, while more sophisticated kernels are available, they do not provide any significant gain in asymptotic bias reduction. Furthermore, they note that, in general, parameter estimates appear to be robust to the choice of kernel.

> 19 This is, in large part, due to two reasons. First, standard automated procedures for bandwidth selection for nonparametric regression have been developed with the estimation of functions at interior points in mind; the nature of the trade-off between bias and variance at interior points may differ from the nature of the trade-off at boundary points (Ludwig and Miller 2005). Second, the literature on the development of bandwidth selectors customized to the RD context is an emergent one (e.g., McCrary and Royer 2006; Imbens and Lemieux 2008; Imbens and Kalyanaraman 2009). In addition, the relative finite sample performances of the proposed bandwidth selection methods for RD designs have yet to be rigorously assessed.

22
acceptably balances restricting estimation to a local neighborhood around the cutoff against having sufficient statistical power to yield informative estimates of treatment effects. Notwithstanding, we examine whether our RD impact estimates and statistical inference results are sensitive to increasing the optimal bandwidth by 50% and 100%, potentially trading off increased bias for reduced variance.

# **5. Data and sample**

# _Data_

_Baseline data_ : The baseline data come from the program application records and SLQAT test records collected and maintained by PEF. In phase 3, properly filled-in application forms received by PEF before the announced deadline of April 2007, as well as forms filled out by schools solicited by PEF inspection teams on their way to inspect applicant schools were considered. In phase 4, only properly filled-in forms received by PEF before the announced deadline of July 2007 were considered. PEF collected 1,070 and 1,430 properly filled-in application forms in phases 3 and 4, respectively.<sup>20</sup> PEF constructed a school-level application electronic database for each phase which contains information on school characteristics (location, gender type, level, physical infrastructure, and registration status), total school enrollment separately by gender, total number of teachers and administrative staff separately by gender, and the minimum and maximum monthly teacher salaries in the school.<sup>21</sup> These databases serve as the source of baseline data on the following outcomes measured at the school-level: number of students, teachers, classrooms,

> 20 The total number of unique applications received by PEF is unknown as all rejected applications were discarded. 21 As part of this inspection, the data provided by the school in the application form are verified. These inspection data would have been useful for checking the accuracy of the application data; they were however collected in paper form and not entered into an electronic database. Consequently, these data are unavailable for the purposes of this study.

23
blackboards, and toilets. The school-level outcomes of student-teacher and student-classroom ratios are constructed by us using the data on total number of students, teachers, and classrooms. These databases also serve as the source of baseline data on school-level covariates, namely location, gender type, level, and registration status.

The SLQAT serves as the source of data on the school’s student pass rate, our treatment assignment variable _z_ . Separate electronic databases have been constructed by PEF for phase-3 and phase-4 SLQAT takers. These databases contain the total score for each student that took the test. In the databases, student test scores are organized by school and, within school, by grade. The school identification information provided in the databases comprise of the school’s name (at times, with some address elements) and location (tehsil<sup>22</sup> and district). Although PEF constructed the SLQAT pass rate for each school, we used the student test score data in order to construct our own measure of school SLQAT pass rates. The match rate between their calculation of school SLQAT pass rates and ours’ in both phases is nearly perfect at 99.5%. While it makes virtually no difference, we use our measure of school SLQAT pass rates as the treatment assignment variable Constructing a single electronic database for our analysis required that the application data are linked with the SLQAT data at the school level. Unfortunately, the same unique school identification variable was not used across databases. Consequently, we linked the application databases to the SLQAT databases using an iterative visual-matching process. First, schools were matched across databases using the district name and school name variables.<sup>23</sup> Exact matching failed in a number of cases as PEF did not maintain consistency in the spelling, word ordering,

> 22 Tehsil is the spatial unit of government administration one tier below district. There are 127 tehsils in Punjab. 23 Although information on the school’s tehsil was also available in both the application and test databases, this information was error-ridden. Consequently, we did not use this information in the cross-database matching exercise.

24
and completeness of the school’s name across the two databases; hence, matching on school name frequently required matching on key words and word patterns. In cases where we suspected that the combination of district name and school name (even with keyword and word pattern matching) did not yield a unique school record in a database, where possible, the set of matching variables was extended to include school address. This extension helped resolved a number of cases where we were tentative. On the basis of this exercise, 94% and 97% of school records in the SLQAT databases were linked with school records in the application databases for phases 3 and 4, respectively. Only the linked school records are used in our analysis, which yield sample sizes of 747 and 830 schools in phases 3 and 4, respectively. We refer to these samples as the SLQAT samples.

_Follow-up telephone interview data_ : We collected follow-up outcome data via telephone in October 2008 from phase-3 and phase-4 schools with SLQAT pass rates between  /  15 percentage points of the cutoff. This information was gathered by a small team of independent interviewers. We were able to collect the data using this mode as schools were required to provide the telephone number of the school owner or administrator when they applied to the program, and the information was included in the school application electronic databases for both phases.

We obtained data on the following outcomes: the number of students, teachers, classrooms, blackboards, and toilets (the data on the last two outcomes were collected only for the phase-4 schools).<sup>24</sup> We also obtained information on the program participation status of the schools, and crosschecked this information against PEF’s records of program schools as of September 2008. The school sample sizes within the selected pass rate range from the cutoff are

> 24 We are unable to speculate on the accuracy of these data vis-à-vis the same data collected through, for example, face-to-face interviews in the field, as we are unclear what context factors might lead to differences in accuracy across survey modes.

25
268 and 319 schools in phases 3 and 4, respectively. These numbers constitute 36% and 38% of the respective SLQAT samples. We refer to these samples as the cutoff neighborhood samples.

The period of the follow-up data collection was roughly 14 and 10 months after the receipt of the first subsidy payment by phase-3 and phase-4 program schools, respectively. The treatment exposure period under investigation roughly covers the second half of the 2007/08 academic year and first half of the 2008/09 academic year.

_Unit nonresponse analysis_ : Despite repeated attempts by the interviewers, not all schools in the cutoff neighborhood samples were contactable over the telephone. Unit nonresponses were mainly due to wrong numbers or unattended telephone calls; when reached, refusals were rare. There were no cases of item nonresponse. Roughly 28% and 22% of schools have missing data due to unit nonresponse in the phase-3 and phase-4 cutoff neighborhood samples, respectively. These large percentages are expected to reduce the study’s statistical power (which does pose some concern given that our study is low-powered to begin with) but do not necessarily introduce sample selection bias.

To investigate the presence of systematic bias, we estimate simple bivariate correlations between unit nonresponse and selected covariates, separately by phase. The covariates that we examine comprise of indicator variables for (initial) treatment assignment (i.e., whether the school’s SLQAT pass rate is above the cutoff) as well as baseline measures of the school’s registration status, level, gender type, and location. Table 7 presents nonresponse counts and rates (Panel A) and the estimated correlations (Panel B). It appears that the probability of nonresponse is not significantly associated with treatment assignment: the estimated correlation coefficients are .03 and .07 and the associated standard errors are .06 and .05 for the phase-3 and phase-4 cutoff neighborhood samples, respectively. In addition, as Columns 3 and 6 in the table 26
show, barring a few exceptions—which could arise simply due to random chance—the probability of nonresponse does not appear to be significantly associated with the selected school covariates. Thus, the evidence suggests that we cannot reject the null hypothesis that nonresponse rates are generally comparable for marginal failers and marginal passers across the selected dimensions measured at baseline.

Even if nonresponse rates are similar between marginal failers and passers, it is still possible that the selection process for nonresponding marginal failers differs from the selection process for nonresponding marginal passers (Duflo et al. 2007). Table 7 also reports bivariate correlations between unit nonresponse and selected covariates measured at baseline separately for marginal failers and marginal passers (Columns 1 and 2 for phase 3 and Columns 4 and 5 for phase 4). The evidence generally suggests that we cannot reject the null hypothesis that the underlying selection processes for nonresponse for both marginal failers and marginal passers are random. Based on these collective findings, we assume that the nonresponse rate is locally smooth in the assignment variable at the cutoff.

# _Sample_

Table 8 presents statistics on the distribution of schools by selected characteristics measured at baseline, separately by phase and for the SLQAT and cutoff neighborhood samples. We highlight two patterns. First, across both phases and samples, the distributional patterns across characteristics are broadly comparable. For example, the majority of schools are (1) middle schools (63–72%), (2) coeducational (82–87%), (3) officially-registered (81–88%) and (4) rural (53–59%). Second, comparing the phase-3 SLQAT sample to the phase-4 SLQAT sample (Columns 1–3), pairwise _t_ -tests of the equality of proportions show significant differences in the 27
distribution of schools across several characteristics. It appears that, among other things, the shares of secondary schools, single-sex schools, urban schools, and registered schools are larger in the phase-3 SLQAT sample than in the phase-4 SLQAT sample. These distributional differences across phases lose their statistical significance when we restrict our attention to the cutoff neighborhood samples (Columns 4–6). Some part of the explanation for the loss in significance is likely due to reduced statistical power resulting from the smaller sample sizes for the cutoff neighborhood samples relative to the SLQAT samples.

Table 9 presents means and standard deviations for selected outcomes measured at baseline, again separately by phase and for the SLQAT and cutoff neighborhood samples. In the phase-3 SLQAT sample (Column 1), we find that schools on average have 253 students; 10 teachers, classrooms, and blackboards; and 3 toilets. Mean student-teacher and studentclassrooms ratios are 25:1 and 28:1, respectively, which, to begin with, are below the stipulated maximums for program benefit maintenance. The differences across phases in the SQLAT samples (Columns 1–3) lose their statistical significance when we restrict our attention to the cutoff neighborhood samples (Columns 4–6). Again, some part of the explanation for the loss in significance is likely to be the lower statistical power of the cutoff neighborhood samples relative to the SLQAT samples. Notwithstanding, the sizes of the inter-phase differences in the cutoff neighborhood samples are generally substantially smaller than the corresponding differences in the SLQAT samples. In sum, the phase-3 and phase-4 cutoff neighborhood samples appear to be similarly composed with respect to mean outcomes measured at baseline.

28
# **6. Results**

Before presenting our RD estimates of the impacts of the FAS program, we discuss results from a couple of model specification tests. First, we show that there is a discontinuous change in the conditional probability of treatment at the SLQAT pass rate cutoff, as the suitability of the RD model hinges on this feature of the data. Second, we examine whether the density function of the SLQAT pass rate exhibits local smoothness at the cutoff. This test is particularly useful if data are unavailable to directly test whether the conditional mean untreated outcomes at the cutoff exhibit local smoothness. However, in our case, we can directly test the identifying condition for the RD design by using our baseline data on outcomes. Thus, we perform the density test mainly to check whether the result of this test is consistent with the result of the test of the identifying condition.

# _Discontinuity in the probability of treatment at the cutoff_

Figure 3 plots the local linear regression functions estimated separately above and below the cutoff, separately for the phase-3 and phase-4 cutoff neighborhood schools _with_ follow-up data on, among other things, program participation status (the optimal bandwidth is six percentage points for both samples). The jump in the probability of treatment is clearly visible for both samples. For the phase-3 sample, as expected, the probability of treatment is one above the cutoff and positive but less than one below (with 42% of marginal failers in phase 3 gaining participation status in phase 4). Furthermore, we observe that the probability of treatment is increasing as we approach the cutoff from below. A plausible and likely explanation for this pattern is that phase-3 marginal failers closer to the cutoff have a greater chance of passing the phase-4 SLQAT and entering the program than marginal failers farther away. For the phase-4 29
sample, as expected, the probability of treatment jumps discontinuously from zero to one at the cutoff. These patterns in the conditional probabilities of treatment motivate our selections of partially-fuzzy and sharp RD designs for the phase-3 and phase-4 SLQAT samples, respectively.

_Local smoothness in the density function of SLQAT pass rates_ Rejection of local smoothness in the density function of the assignment variable at the cutoff may suggest manipulation of the assignment variable _z_ (McCrary 2008). Manipulation is plausible in our case as the SLQAT pass rate cutoff for program participation was advance public knowledge, and schools and perhaps even program managers may have an interest in a particular treatment outcome. The pass rate that the school _actually_ received on the SLQAT may be subject to partial manipulation, which is typically benign.<sup>25</sup> However, PEF staff exercise full control over the calculation of SLQAT pass rates, as well as their reporting, thus, introducing the risk of complete manipulation.<sup>26</sup> Given that we use SLQAT pass rates calculated by us based on the individual student test score data as our measure of the assignment variable, and these calculations nearly always match the pass rates calculated by PEF, manipulation would have to occur at the individual student level. Manipulation at this level is less straightforward; nonetheless, we cannot rule out the possibility.

Figures 4 and 5 present frequency histograms of schools with bins equal to the integer values of SLQAT pass rates for the SLQAT and cutoff neighborhood samples, respectively, separately by phase. A simple visual inspection of the SLQAT pass rates for both the phase-3

> 25 The day the SLQAT is offered, the classes which are tested, and the specific SLQAT papers offered at the school are stochastic elements from the perspective of the school and outside its control.

> 26 PEF staff may face incentives to manipulate the pass rates of marginal passers downwards (to just below the cutoff) if, for example, the program budget limits the number of schools that the program can be offered to. Given that PEF reports that they were flush with funds during this period, we can probably discount downward manipulation due to this reason. Alternatively, sympathetic PEF staff might manipulate the pass rate of marginal failers upwards (to just above the cutoff), so that these schools become eligible to participate in the program.

30
and phase-4 samples suggests the presence of discontinuities in their densities in the immediate neighborhood of the cutoff. This is confirmed by formally testing for discontinuities in the densities at the cutoff by applying a test proposed by McCrary (2008) involving separate kernel density estimations below and above the cutoff.<sup>27</sup> However, this pattern is not unique to the cutoff location; the histograms show multiple sharp frequency peaks and troughs (at times, at adjacent pass rates) across the SLQAT pass rate support.

# _Local smoothness in conditional mean outcomes at baseline_

As a direct test of the identifying assumption, we examine whether mean outcomes measured at baseline satisfy local smoothness at the cutoff using two approaches: by (1) examining differences in simple means between marginal failers and passers for alternative cutoff neighborhood sizes and (2) estimating the discontinuity in the conditional means at the cutoff via local linear regressions at the selected alternative bandwidths. Table 10 presents the simple mean outcomes at baseline for marginal failers and differences from these means for marginal passers for two cutoff neighborhood sizes:  / 15 percentage points and  / 5 percentage points. The evidence suggests that, for both phase 3 and 4, by and large, there are no statistically-significant differences between marginal failers and passers in mean outcomes measured at baseline.

Table 11 presents sharp RD estimates of baseline mean outcomes at the cutoff using local linear regressions, separately by phase and for the selected alternative bandwidths (the optimal bandwidth is four and three percentage points for the phase-3 and phase-4 cutoff neighborhood samples, respectively). In addition, Figures 6a–6c depict the estimated local linear regression functions for the phase-3 and phase-4 cutoff neighborhood samples, using their respective optimal bandwidths. In general, the evidence suggests that we can reject local smoothness in 27 Results from the tests are available from the authors upon request.

31
baseline mean outcomes at the cutoff more frequently than what our earlier simple comparisons of baseline mean outcomes between marginal failers and passers would have led us to state, as well as more frequently than we would have expected by random chance given standard significance levels.

Out of the 42 individually-estimated RD parameters across phases, outcomes, and selected bandwidths, seven estimates (17%) are statistically-significant. Virtually all the statistically-significant RD estimates are negative in sign, suggesting that program schools tend to be smaller and have lower levels of inputs at baseline. The inference results are however not robust to the selected alternative bandwidths. Given this sensitivity in inference, we consider the results to generally satisfy the identifying condition.

# _RD impact estimates_

Table 12 presents our RD impact estimates based on local linear regressions, separately by phase and for the selected alternative bandwidths (the optimal bandwidth is six and three percentage points for the phase-3 and phase-4 cutoff neighborhood samples, respectively). Figures 7a–b plot the estimated local linear regression functions using the optimal bandwidth choice for the phase4 cutoff neighborhood sample.

For the phase-3 cutoff neighborhood sample (Columns 1–3), whose data are subject to a partially-fuzzy RD design, we find no evidence of significant program impacts across the outcomes of interest. The magnitudes of the impact estimates also appear to be sensitive to the selected alternative bandwidths. In addition, the empirical standard errors associated with the impact estimates are inordinately large, suggesting weak identification.

32
For the phase-4 cutoff neighborhood sample (Columns 4–6), whose data are subject to a sharp RD design, we find evidence of significant positive impacts on the number of students, teachers, classrooms, and blackboards in marginal passers; furthermore, their significance is robust to the selected alternative bandwidths. The accompanying figures show a discernible structural change in the mean levels for these outcomes marginally above and below the cutoff. However, as with the partially-fuzzy RD estimates for the phase-3 cutoff neighborhood sample, the magnitudes of the impacts appear to be sensitive to bandwidth. In particular, the estimates drop fairly sharply when we increase the bandwidth from the optimal choice of 3 percentage points by 50% to 4.5 percentage points. The most conservative impact estimates at the cutoff on the number of students, teachers, classrooms, and blackboards are 85, 3.4, 4, and 2.8, respectively. Relative to the baseline means for these outcomes in the phase-4 cutoff neighborhood sample, the absolute impact estimates translate into percent impact estimates of roughly 37%, 37%, 47% and 27%, respectively. These impacts are substantial. They are particularly impressive given the short treatment exposure under investigation: the follow-up telephone interview data were collected only some ten months after phase-4 program schools received their first subsidy payment.

Notwithstanding, we do not find evidence of impacts at the cutoff on the number of toilets and student-teacher and student-classroom ratios. This finding is robust to the selected alternative bandwidths. The lack of evidence of a positive effect on the number of toilets is concerning given the large expansion in enrollment in marginal passers and its potential negative bearing on the use and maintenance of the facility. The finding may reflect that PEF does not encourage the adequate provision of this facility to the same degree as the provision of other types of physical infrastructure (such as classrooms). Given the expansion in enrollment in 33
marginal passers, the absence of effects on the ratios suggests that program schools have expanded the number of teachers and classrooms (which we find) in lock-step with this expansion. This behavior may be driven in large part by PEF’s condition that program schools maintain these ratios below stipulated levels.

# _Falsification test: RD estimates at false cutoffs_

Table 13 presents sharp RD estimates for our outcomes of interest based on local linear regressions for the phase-4 cutoff neighborhood sample at two arbitrarily-selected false cutoffs: 57% and 77%. The false cutoffs are equidistant from the true cutoff of 67%. We expect to find local smoothness in the conditional mean outcomes at these cutoffs. The subsample for the investigation at 57% is schools with SLQAT pass rates between 52% and 66%. Likewise, the subsample for the investigation at 77% is schools with SLQAT pass rates between 67% and 82%. The optimal bandwidths for the lower and upper samples are six and three percentage points, respectively. At both cutoffs, the evidence suggests that we cannot reject the null hypothesis of local smoothness in the conditional mean outcomes. The inference results are robust to the selected alternative bandwidths.

# _Cost-effectiveness estimates_

We estimate the cost-effectiveness of the program in relation to enrollment gains using two alternative approaches. Using the conservative estimate of the sharp RD impact on enrollment for the phase-4 cutoff neighborhood sample (85 students), a first way that we estimate the program’s cost-effectiveness is by deriving the annual rupee cost of one additional student in a program school _induced_ by the program. Given a baseline mean school size of 232 students for 34
schools in the phase-4 cutoff neighborhood sample (which we treat as the number of children that would have attended the program school in the absence of the treatment) and an annual subsidy amount of 3,600 rupees (US$42.4) per student,<sup>28</sup> it costs 13,426 rupees (=3,600  (232+85)/85) (US$157.7) to induce an additional student per year. In comparison, this cost is roughly half of the cost of inducing an additional student per year through conditional cash transfers to female students in public secondary schools in Punjab (Andrabi et al. 2008a).

Following the approach by Evans and Ghosh (2008), a second way we calculate the program’s cost-effectiveness is by deriving the annual per-student cost of increasing enrollment in program schools by 1%. Using an annual subsidy amount of 3,600 rupees per student and the sharp RD impact estimate on enrollment of 37%, we estimate a cost-effectiveness ratio of 97 rupees (=3,600/37) (US$1.1). This estimate compares extremely favorably with the estimated cost-effective ratios of other evaluated education interventions across the developing world which generated enrollment gains as reported in Evans and Ghosh. In fact, the FAS program’s ratio ranks among the very lowest.<sup>29</sup> What is more, our cost-effectiveness estimate surprisingly neighbors Evan and Ghosh’s estimate of the cost-effectiveness of the per-student subsidy program in Balochistan, Pakistan evaluated by Kim et al. (1999) and noted in Section 1.

# **7. Conclusion**

In this paper, we estimate the causal effects of the FAS program, a recently-instituted public subsidy program to low-cost private schools, in the province of Punjab, Pakistan on student enrollment and schooling inputs. Our findings on the impacts of the program differ by phase. For

> 28 The annual subsidy amount per student is roughly equal to the annual _program amount_ per student, as the perstudent amounts for program administrative costs and the teacher and school bonuses add less than 1% to the amount. This is principally due to the large number of students currently covered under the program. 29 This result remains qualitatively unaltered if we precisely follow the currency conversion and inflation adjustments steps taken by Evans and Ghosh to fix all ratios in 1997 US dollars.

35
phase-3 SLQAT takers, applying a partially-fuzzy RD design, we find no evidence of program impacts at the cutoff on our outcomes of interest. We posit that this is likely due to weak identification arising from a small jump in the probability of treatment at the cutoff. In contrast, for phase-4 SLQAT takers, applying a sharp RD design, we find robust evidence that the program significantly increased the number of students, teachers, classrooms, and blackboards in marginal passers. The impact estimates at the cutoff were sizeable: our conservative estimates indicate that the program expanded marginal passers by roughly 85 students, and 3–4 teachers, classrooms, and blackboards. These impact estimates are also large relative to the mean baseline values for these outcomes for phase-4 SLQAT takers near the cutoff. What is more, they are particularly impressive given that phase-4 program schools were exposed to only about ten months of treatment before the follow-up telephone interview data were gathered. Finally, our cost-effectiveness estimates suggest that the program is among the cheapest interventions in developing countries for inducing enrollment gains.

Two potential threats to the internal validity of our impact estimates are however present. These threats arise from specific design and implementation features of the program. The first threat is program spillovers to non-program schools (of which marginal failers are but a specific subset) that operate in the same local schooling markets as program schools. Design features of the program such as the free-schooling condition for program benefit maintenance can alter the terms of local market competition, providing program schools with a competitive edge vis-à-vis non-program schools. The altered terms could result in impact estimates being upwardly biased if, for example, they induce the flow of students (and accompanying them, teachers) from nonprogram to program schools leading to the shrinking (or, at an extreme, the shutdown) of nonprogram schools and/or they discourage investments in the quantity and quality of inputs. On the 36
other hand, the altered terms could result in impact estimates being downwardly biased if they induce non-program schools to adapt to more effectively compete in the new local market environment created by the program such as by ratcheting up their investments in the quantity and quality of inputs and/or altering their fee structures to retain their existing student bodies and attract new students. Both types of effects could be present simultaneously; consequently, the direction of the net effect is theoretically ambiguous.

The second threat arises from anticipation of future treatment. Program entry is not a oneoff event. To date, there have five calls for applications over a three-year period. Given this pattern, it is conceivable that non-program schools interested in joining the program might alter their behavior in anticipation of a future call for applications and in an effort to increase the likelihood of program entry. These actions could, for example, take the form of non-program schools investing in more and better quality inputs and resources. Anticipation in this case would result in impact estimates being downwardly biased. In particular, behavioral changes due to anticipation might be most applicable to non-program schools who just failed to achieve the SLQAT pass rate cutoff in an earlier phase of entry (our marginal failers), as we would expect that the marginal costs of investments and efforts required for failers from a previous phase to gain program entry are likely to be decreasing as one approaches the cutoff from below.

While when the RD model is applied to the right data designs, it yields internally-valid estimates, the generalizability of these impact estimates is likely to be limited as they are in principle only valid for narrowly-defined subpopulations. In the case of the phase-4 SLQAT taker data, which are fitted using a sharp design, the RD impact estimates are valid for low-cost private schools that successfully applied to the program, cleared the physical inspection, and obtained pass rates near the SLQAT cutoff. Even if we generalize our estimated treatment effects 37
to be average treatment effects over the full range of SLQAT pass rates, the further generalizability of the estimated effects is limited by sample selection, as the SLQAT pass rate is only available for low-cost private schools in the seven main program districts that learned about the call for applications, applied to the program, passed the inspection screening, and took the SLQAT. This group of schools may not be representative of the population of low-cost private schools in the program districts, leave alone the population of low-cost private schools more widely. Given what we know about the steps that precede taking the SLQAT, the extent to which the population of SLQAT takers diverges from the broader population of low-cost private schools in the program districts is largely determined by how demanding PEF’s physical inspection screening is. Evidently, this screening has bite: as mentioned before, only 75% and 61% of schools that were inspected by PEF cleared it and took the SLQAT in phases 3 and 4, respectively.

38