metadata: authors: 0: author: Chaudhury,Nazmul
1: author: Parajuli,Dilip
count: Pakistan
docna: 0: docna: Conditional cash transfers and female
            schooling : the impact of the female school stipend program
            on public school enrollments in Punjab, Pakistan
docty: Policy Research Working Paper
subtopic: Educational Sciences,Educational Institutions & Facilities,Gender and Development,Employment and Unemployment,Effective Schools and Teachers
teratopic: Education,Gender
abstracts: cdata!: Instead of mean-tested conditional cash
            transfer (CCT) programs, some countries have implemented
            gender-targeted CCTs to explicitly address intra-household
            disparities in human capital investments. This study focuses
            on addressing the direct impact of a female school stipend
            program in Punjab, Pakistan: Did the intervention increase
            female enrollment in public schools? To address this
            question, the authors draw on data from the provincial
            school censuses of 2003 and 2005. They estimate the net
            growth in female enrollments in grades 6-8 in stipend
            eligible schools. Impact evaluation analysis, including
            difference-and-difference (DD), triple differencing (DDD),
            and regression-discontinuity design (RDD) indicate a modest
            but statistically significant impact of the intervention.
            The preferred estimator derived from a combination of DDD
            and RDD empirical strategies suggests that the average
            program impact between 2003 and 2005 was an increase of six
            female students per school in terms of absolute change and
            an increase of 9 percent in female enrollment in terms of
            relative change. A triangulation effort is also undertaken
            using two rounds of a nationally representative household
            survey before and after the intervention. Even though the
            surveys are not representative at the subprovincial level,
            the results corroborate evidence of the impact using school
            census data.
geo_reg_key: 517198,517212,517191
repnme: repnme: Conditional cash transfers and female
            schooling : the impact of the female school stipend program
            on public school enrollments in Punjab, Pakistan
display_title: Conditional cash transfers and
            female schooling : the impact of the female school stipend
            program on public school enrollments in Punjab, Pakistan
listing_relative_url: /research/2006/12/7289481/conditional-cash-transfers-female-schooling-impact-female-school-stipend-program-public-school-enrollments-punjab-pakistan
topicv3: Teachers Management m1326119 1523,Education m1327048 470,Gender m1327621 615,Census Data m1504073 2763,Stipend m1583603 4402
publishtoextweb_dt: 2006-12-22T00:00:00Z
historic_topic: Education,Gender
no_of_pages: 37
author_keys: 0: author_key: 455781
1: author_key: 452322
model_extractions: WPS4102 <mark>I</mark> MPACT <mark>E</mark> VALUATION <mark>S</mark> ERIES <mark>N</mark> O <mark>. 9</mark>

**Conditional Cash Transfers and Female Schooling:**

**The Impact of the Female School Stipend Program on Public School**

**Enrollments in Punjab, Pakistan**

Nazmul Chaudhury Dilip Parajuli The World Bank

# Abstract

Instead of mean-tested Conditional Cash Transfer (CCT) programs, some countries have implemented gender-targeted CCTs to explicitly address intra-household disparities in human capital investments. This study focuses on addressing the direct impact of a female school stipend program in Punjab, Pakistan – Did the intervention increase female enrollment in public schools? To address this question, we draw upon data from the provincial school censuses 2003 and 2005. We estimate the _net_ growth in female enrollments in grade 6-8 in stipend eligible schools. Impact evaluation analysis, including difference-and-difference (DD), triple differencing (DDD) and regression-discontinuity design (RDD), indicate a modest but statistically significant impact of the intervention. The preferred estimator derived from a combination of DDD and RDD empirical strategies suggests that the average program impact between 2003 and 2005 was an increase of six female students per school in terms of absolute change and an increase of 9 percent in female enrollment in terms of relative change. A triangulation effort is also undertaken using two rounds of a nationally representative household survey before and after the intervention. Even though the surveys are not representative at the sub-provincial level, the results corroborate evidence of the impact using school census data.

# World Bank Policy Research Working Paper 4102, December 2006

_The Impact Evaluation Series has been established in recognition of the importance of impact evaluation studies for World Bank operations and for development in general. The series serves as a vehicle for the dissemination of findings of those studies. Papers in this series are part of the Bank’s Policy Research Working Paper Series. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ __________________________________ The authors would like to thank Jishnu Das, Michelle Riboud, Tahseen Sayed and an anonymous referee for their valuable comments and suggestions on the paper.
# **1. Introduction**

Conditional Cash Transfer (CCT) programs targeted to poor households are rapidly becoming a key policy instrument used by developing countries to reduce poverty and increase human capital investments (Janvry and Sadoulet 2006). They have ranged from employing poor adults conditioned upon their labor contributions towards public infrastructure construction (e.g., Maharashtra Employment Guarantee Scheme in India), to giving poor households in-kind food grain distribution conditioned upon sending their children to school (e.g., Food for Education Program in Bangladesh), to giving poor households cash conditioned upon school attendance of their children (e.g., Bolsa Escola in Brazil), to giving poor households cash and nutritional supplements conditioned upon sending their children to school and taking them to health clinics for check-ups (e.g., PROGRESA/Oportunidades in Mexico). Targeting poor households is, however, a non-trivial endeavor in many developing countries which often do not have systematic household/individual level income or consumption data (van de Walle 1998). Proxy means testing can be used as an alternative targeting method but its use so far has been largely limited to middle-income countries because it is administratively challenging (Coady, Grosh, and Hoddinott 2004). Furthermore, in many developing countries households under-invest in female education and health due to complex socio-economic factors, often necessitating gender-targeted interventions (Ezeminari, Chaudhury, and Owens 2000)<sup>1</sup> .

Instead of mean-tested CCTs, some countries have implemented gender-targeted CCTs to explicitly address intra-household disparities in human capital investments. One of the most prominent examples of this genre of CCTs is the Female Secondary School Stipend (FSSS) Program in Bangladesh. On one hand it is a typical demand-side CCT which gives money to girls conditioned upon their enrollment in secondary school (among other conditions such as maintaining regular attendance and passing grades; remaining unmarried till 18). On the other hand it goes beyond most CCTs by explicitly factoring in supply-side constraints by inducing the growth of private secondary schools in rural areas by providing government financing conditioned upon the ability of the school to attract female pupils. While it is almost impossible to adequately evaluate the impact of FSSS given the absence of randomization or a properly defined control and treatment group with baseline and follow-up data, (due to rapid scaling-up of the program), the program was associated with a rapid increase in the share of female secondary

> 1 Means-tested CCT programs like PROGRESA/Oportunidades pay higher cash transfers for female children, thereby addressing issues of gender disparity within a poverty-targeted framework.

2
school enrollment. The share of female enrollment went from less than 35% before the program, to more than 50% in less than a decade after the program. The program might actually have been ‘too-successful’, resulting in a reversal where we now possible have under-investment in male secondary schooling within the household (Khandker and Pitt 2003; Asadullah, Chaudhury, and Dar 2006). The FSSS program in Bangladesh has inspired a similar program in Punjab, Pakistan, with some important differences.

# **2. Background of the CCT Program and Evaluation Objectives**

Unlike Bangladesh where the public education system is highly centralized, Pakistan has essentially devolved responsibility over public education to the provincial level. The focus of our study is Punjab, the largest and wealthiest province in Pakistan, where literacy and enrollment rates are higher and gender differences are lower compared to other provinces. Despite being the leading province in Pakistan in terms of educational outcomes, the provincial government (Government of Punjab, henceforth referred to as GoP) was concerned with a variety of problems plaguing the education sector - insufficient allocation of resources to the education sector, systemic weakness in public sector delivery as a result of over-centralization and inadequate management, and poor performance of the schooling system in terms of quality, access and governance (The World Bank, 2004)<sup>2</sup> . To address these issues, in 2003 GoP initiated the most ambitious reform program ever in Pakistan – the Punjab Education Sector Reforms Program. The Punjab Education Sector Reform Program (PESRP), also financed via World Bank lending, has three pillars: (A) public finance reforms to realign expenditures at the provincial and district level toward education and other pro-poor expenditures; (B) devolution and public sector management reforms; and (C) education sector reforms to improve quality, access, and governance of the education system.

# _Reforms within the Education Sector_

Within the education sector, the reform program has introduced a number of interventions: distribution of free textbooks, upgraded/new school infrastructure, recruitment of regular and contract teachers, increasing entry qualifications of teachers, female stipend program, providing greater authority to school councils, NGO and community management of school councils. Not

> 2 Program Document for Education Sector Adjustment Credit for the Government of Punjab Province, January 8, 2004.

3
all of these reform components have been phased in at the same time; certain components are yet to be fully implemented (e.g., enhanced community involvement in school councils).

# _Female Secondary School Stipend Component_

This component was first implemented in February of 2004. Fifteen of Punjab’s 34 districts were selected as program districts on the basis of average literacy rate of population 10 years and older as estimated from Population Census 1998 data. The cut-off literacy rate was 40 percent: 15 selected districts below the cut-off were stipend-eligible and the remaining 19 above the cut-off were not (Table 1 shows district level literacy rates). Randomization of stipend districts was not politically feasible, nor deemed to be equitable. Rather, transparent allocation of this component to the lowest-literacy districts was considered to be the least contentious way to introduce this new stipend program<sup>3</sup> .

Under the CCT program the girl receives a stipend conditional on her being enrolled in grade 6-8 in a government girl’s school<sup>4</sup> in a target district and conditional on her maintaining average class attendance of at least 80 percent. Eligible student receives Rs.200 per month<sup>5</sup> . For households which were going to send their daughters to middle/secondary school anyway, it is essentially an income transfer. Only for households with out-of-school daughters, is it an incentive payment. Funds are transferred directly to the student (student’s household) via postal money order from the Education District Office (EDO), which has a special account at the Provincial Program Monitoring and Implementation Unit (PMIU) of the Education Department. An important difference with the Bangladesh FSSS program is that females are eligible for the stipend _only if they attend public secondary schools_ . Indeed the secondary education sector has a dearth of private schools (unlike the primary education sector), however, linking the stipend program to only public schools highlights the initial public focus of the reform agenda. One of the long-run objectives of the stipend program targeted to females is to increase the supply of female teachers and other female service providers (e.g., health care staff), particularly in low literacy and relatively poorer regions of Punjab (most of the stipend districts fall in southern Punjab). Interestingly, Andrabi, Das, and Khwaja (2006), find that female graduates of public high schools

> 3 Implementation of the stipend program within the 15 low literacy districts could have been randomly phased in, however, there was no variation in program implementation within the target districts either. 4 All public secondary schools in Punjab are segregated by gender.

> 5 In dollar terms, this is little over $US 3 per month. Household survey data from Punjab province (PIHS 2001-02) estimates that per student out-of-pocket cost of middle school education in a public school averages Rs. 160 per month. Median household consumption expenditure from the same data is Rs. 5,900 per month.

4
might have been instrumental the in staffing of private primary schools and, thereby contributing to the growth of the private sector.

In the PESRP program document<sup>6</sup> , it notes that: a) female middle school enrollment rate increased from 43% (baseline 2003) to 53% in 2005, b) share of female enrollment in government primary and middle school increased from 45% in 2003 to 50% in 2005, and c) female dropout rates between grade 5 and 6 decreased by 25%, and that in middle school decreased by 20%. While these performance indicators point towards an increase in enrollments and decrease in dropouts, none of them specify the intended net impact in a counterfactual framework (in program districts relative to non-program districts, or in girls schools relative to boys schools). Furthermore, given that post and pre reform school census data forms the basis of these figures (rather than household-based data), combined with the fact that school census data do not have individual student identifiers, the dropout rates should be taken with several grains of salt, at best.

Given that PERSP covers the entire province, evaluating the overall impact of the reform on enrollments becomes quite difficult in the absence of a suitable counterfactual. The State Bank of Pakistan (2006)<sup>7</sup> , SBP, recently conducted an impact analysis of the Punjab Education Reform program using the entire province as the treated unit and the other three provinces of Pakistan as counterfactuals. The analysis is based on a difference-in-difference estimator of progress in gross and net enrollment indicators reported in the Pakistan Social and Living Standards Measurement survey (2004-05). They conclude that the program contributed significantly to improving gross and net enrollments and that suggest that extension of similar program activities elsewhere in the country would substantially improve the country’s education access outcomes. Because SBP analysis attempts to evaluate the overall PERSP program and not any specific intervention such as the stipends program, it is difficult to attribute its usefulness to the impact analysis of girls’ stipends program. While their use of relevant data and technique is a noteworthy start, the choice of other provinces as counterfactuals (controls) is problematic because of significant social, political, and institutional differences between Punjab and other provinces. Moreover, their analysis does not include schooling outcome indicators for middle school grades– the relevant indicator for female stipends program. Precise estimates (proper statistical inference such as including standard errors) of these net impacts are missing in their analyses.

> 6 _Annex B_ of Program Document for Education Sector Adjustment Credit for the Government of Punjab Province, January 8, 2004.

> 7 State Bank of Pakistan (2006), Special Section: Impact Analysis of Punjab Education Sector Reforms in _First Quarterly Report for FY06, The State of Pakistan’s Economy_ .

5
# _Evaluation and Policy-Relevant Questions_

This study represents the first systematic effort to evaluate the impact of the female stipend component of PERSP. There are several policy-relevant research questions regarding this stipend program<sup>8</sup> : 1) What is the impact of the stipend component on female enrollment in public middle/secondary schools?

2) What is the impact on overall female enrollment, public and private schools? (Despite the fact that there are fewer private middle schools in the province, there might have been some switching from private to public school.) 3) What is the impact on enrollments across household welfare quintiles? And was the stipend sufficient to attract a female child at the margin (indifferent between attending and not attending)?

4) Did it have any (negative or positive) externality effect on male schooling (via changes in the explicit/implicit cost of male and female schooling within the household)?

5) What is the impact on quality of learning?

6) Does the stipend program have impacts outside of schooling outcomes such as that on health, in particular, on marriage, fertility and malnutrition?<sup>9</sup> 7) Could the money currently being spent on the stipend component be rather spent on other types of interventions with greater impact?

This study focuses on addressing the first question and most direct impact of the intended intervention – did it increase female enrollment in public schools? To address this question, we draw upon data from the provincial Education Management Information Systems (EMIS) school censuses 2003 and 2005. We estimate the _net_ growth in female enrollments in grade 6-8 in stipend eligible schools. Impact evaluation based on counterfactual analysis using methods including difference-and-difference (DD), triple differencing (DDD) and regression-discontinuity design (RDD), indicate modest but statistically significant impact of the intervention. The preferred estimator derived from a combination of DDD and RDD empirical strategies suggests

> 8 This is not meant to be an exhaustive set of policy relevant questions.

> 9 A recent study from Bangladesh indicates that the female secondary education may have had significant positive effects on fertility choices and their children’s malnutrition (A paper presented at a conference on “Towards a Strategy for Achieving the MDG outcomes in Bangladesh” in June 2006 by Qaiser Khan, Syed Rashed Al Zayed, and Yaniv Stopniztky).

6
that average program impact between 2003 and 2005 was an increase of 6 female students per school in terms of absolute change and an increase of 9 percent female enrollment in terms of relative change. A triangulation effort of our findings using school census data is also corroborated with evidence of the impact of the stipend using two rounds of a nationally representative household survey before and after the intervention.

# **3. Methodology**

## _Outcome of interest and unit of analysis_

Frequently used indicators of access to education such as gross or net enrollment rates at a particular level of schooling (e.g. middle school grade 6-8) or school participation rate for a particular age-group (e.g. 10-14 year olds) by geographic regions (e.g. districts) are best estimated using representative household surveys or population censuses. The last Population Census in Pakistan was conducted in 1998. In terms of large-scale household surveys, the two relevant ones are: Pakistan Integrated Household Survey 2001-2002 (PIHS 2001-02) for the preintervention period, and Pakistan Social and Living Standard Measurement survey (PSLM 200405) for the post-intervention period. Unfortunately the sample size and statistical representation for sub-provincial level analysis limited in these surveys – hence, we can only draw upon them to supplement our impact analysis.

The primary data for this study available to us come from the PMIU’s government school census for 2003 to 2005. These school censuses provide school-level information on enrollments by grade for both pre-program and post-program periods. Since public schools in Pakistan are either boys only or girls only, schools can conveniently form the units of our analysis<sup>10</sup> . This implies that our impact estimate on school enrollments can be measured in terms of average growth in grade 6-8 enrollments in program schools relative to non-program schools. Program effect can be interpreted in the following manner: when eligible schools offered stipends to girls, did these schools record an _extra increase_ in enrollments in grade 6-8 relative to non-program schools? We will measure changes in enrollments in terms of both _absolute_ and _percent growth_ . While absolute growth estimates change in actual size of student enrollment, percent growth would estimate change relative to some initial figure. When there is a wide variation in initial middle

> 10 We also estimate program impacts using districts as the unit of observation in one of our sensitivity checks (see section 5).

7
school enrollment numbers, as is the case for Punjab schools under consideration, percent growth often provides alternative meaningful information. The usual comparison group for girl-schools in program districts would be girl-schools in non-program districts. An alternative counterfactual group would be boy-schools in program-districts since boys do not receive stipends regardless of their school location.

# _Empirical strategy_

In an ideal world of true counterfactual impact analysis, net impact of the stipend program would be the difference in the relevant outcome indicator (e.g. growth in school enrollment) in _presence_ of the intervention (treatment) and that in _absence_ of the intervention (control). However, one never observes a unit of evaluation (individuals or schools or districts) in both states (presence of and absence of program) at the same time, thus leading to evaluation problem of missing data. A naïve method to address such missing data problem is to focus on changes (or growth) in school enrollments over time _within_ stipend districts only. This type of estimator can simply overestimate the impact – enrollments could have increased in those districts even in the absence of the stipend program.

# _Difference-in-difference approach_

A more suitable approach is to measure changes in girl enrollments in both stipend and control districts over time and take the difference in changes between the two groups. This so-called difference-in-difference (DD) (or double difference) estimator assumes enrollment growth in nonprogram schools as the counterfactual indicator. DD estimator takes care of the time-trend sensitivity present in before-after estimator by sweeping out time-invariant effects on school enrollments and thereby netting out any growth that could have occurred even in the absence of the stipend program (Ravallion, 2005).

There are in fact three sets of potential control groups we can use in our analysis: girl schools in non-program districts (control 1), boy-schools in program districts (control 2) and non-program districts (control 3). We will use all three control groups when using triple-differencing method, DDD). The following matrix in Table A clarifies this point: 8
Table A:

|_Comparison groups_|Stipend districts|Non-stipend districts|
|---|---|---|
|Girl schools|Treated|Control 1|
|Boy schools|Control 2|Control 3|
|_Outcome indicators_|Stipend (s)|Control (c)|
|Before-program period (b)|Ys,b|Yc,b|
|After-program period (a)|Ys,a|Yc,a|

Net Program impact is then given by ( _y s_ , _a_ − _y s_ , _b_ ) − ( _yc_ , _a_ − _yc_ , _b_ ) = enrollment growth for stipend (treated) group ( _s)_ between before year ( _b)_ and after year ( _a)_ minus enrollment growth for control (non-stipend) group _(c)._ For two periods of data, this can be estimated using the convenient regression<sup>11</sup> : where Δ denotes growth in enrollments for school _i_ , _S_ is the categorical variable that takes a _yi_ value 1 for stipend-receiving school and 0 otherwise. The coefficient α measures the net impact of the stipend program, while the standard error of the coefficient would provide the statistical significance.

The crucial identification assumption in DD estimator is that outcomes in treated and control groups would have followed a parallel path in the absence of the program. In other words, growth in enrollments would have been the same in both groups has there been no stipend program introduced.

However, this assumption is unlikely to be satisfied, thereby making our simple DD estimator biased. The fact that program was not randomly assigned but targeted to low literacy districts

> 11 Alternatively one could use this regression specification _yit_ = μ +γ _S i_ + δ _Tt_ + α _S i_ * _Tt_ + _eit_ where time period dummy is denoted by (T=1 for t=2 and T=0 for t=1). α , now the coefficient on interaction of Time dummy and Stipend dummy, would still be the net impact of the program. Because of panel nature of data and presence of serial correlation, one would need to use cluster (at the individual unit, i.e. school id code in this case) option to get correct standard errors (see Bertrand et al. (2004)).

9
implies that the targeting criteria itself may influence subsequent growth rates, leading to new form of selection bias (Ravallion, 2005)<sup>12</sup> . Because stipend districts are less literate (targeting basis) and possibly less well-off, their initial conditions may very well result in smaller growth in enrollments compared to non-stipend districts that are relatively better-positioned for higher growths (Jalan and Ravallion (1998) illustrate this point in their impact assessment of poor-area development programs in China). In econometric terminology, equation (1) suffers from an endogeneity problem where stipend status S is correlated with the error term e, making the estimated alpha coefficient inconsistent.

# _Alternative DD estimator_

There are a number of ways to deal with this. One could choose boy schools in stipend-districts as control group for girl schools in stipend-districts. This is still a DD approach, except now that the counterfactual is the growth in enrollment in boy schools (that are not receiving stipends) in stipend districts. This would likely alleviate the problem of initial district welfare and literacy conditions that affect subsequent growth. Having treatment and control groups in the same areas would make them more comparable from this standpoint. However, long standing gender preferences (“discrimination”) in the country and subsequent differential improvements in their educational attainments over time makes it very restrictive to assume that enrollment growth of boys and girls would have been similar in the absence of the program. The fact that stipends are targeted to girls implies that their initial (socio-cultural) conditions are not comparable.

# _Difference-in-difference-in-Difference estimation (DDD)_

Because we have data for boys and girls in stipend and non-stipend districts, we can use a method that involves using DD estimate for girls (stipend districts vs non-stipend districts) and DD estimate for boys (stipend districts vs non-stipend districts) and take the difference. In essence, this entails triple differencing (DDD): DD estimate for girls minus DD estimate for boys. Such DDD estimator still allows one to compare girl enrollment growth in stipend relative to nonstipend districts but also provides a cleaner way to “separating out” some of the bias from the differential growth effects that arise due to gaps in initial literacy rates. There are other programs (such as infrastructure improvement) in effect across the entire Punjab province with similar objectives (e.g. to increase overall school participation for both boys and girls) but differential treatment effects of such programs (between low and high literacy districts) will also be

> 12 The selection bias would not have been a so much of a problem if it were time-invariant and additive because DD technique sweeps away this bias by taking changes over time between two groups.

10
accounted for in this triple differencing strategy. DDD is given by ( Δ _yFS_ − Δ _yFC_ ) − ( Δ _yMS_ − Δ _yMC_ ) where subscripts FS denotes female school in stipend district, MC male school in non-stipend control districts and so on. ( Δ _yFS_ − Δ _y FC_ ) is our usual DD estimator of net growth of female enrollment in stipend vs non-stipend districts, ( Δ _yMS_ − Δ _yMC_ ) takes care of differential growth in absence of the program and presence of alternative programs, ( Δ _y FS_ − Δ _yMS_ ) sweeps out any common alternative program effect in stipend districts only (as long as these alternative programs are not gender-targeted or have differential effects by gender). This can be estimated by using our convenient regression framework: Where S is the stipend district (1,0) and F is the female schools (1,0). α _DDD_ , the coefficient on the interaction of Female and Stipend categorical variables, provides us with DDD impact estimate.

# _Other Covariates_

Some of the bias in the estimated impact can also be alleviated through addition of covariates in the DD or DDD regression specifications of equations (1) and (2) respectively. There are broadly three sets of characteristics that affect growth patterns in public schools.

<u>First</u> is the school attributes themselves. Variables such as location (rural/urban), availability of proper water supply, electricity, toilet facilities, and boundary-walls provide some indication of school infrastructure quality. Student-teacher ratio (total students divided by total teachers at work) can either signal quality or absorption capacity: smaller ratio may mean better learning but also higher intake capacity without putting too much burden on teachers.

<u>Second</u> is the presence of private school (for middle school grades) in the area. Andrabi, Das and Khwaja (2005)<sup>13</sup> report that private schooling (share of private sector in total enrollment) in Pakistan grew significantly in the nineties, especially in Punjab province, and the increase was recorded in rural areas and among middle-class and poor population groups. It is not obvious

> 13 Private Schooling: Limits and Possibilities. <u>http://www.economics.pomona.edu/Andrabi/Research/</u>

11
whether an area with a larger share of private sector in total enrollments (for girls in middle school grades) may have a positive or negative impact on growth in public enrollment when an incentive such as the stipend program comes into effect. But Andrabi et al. observations that private schools are located in larger villages with better infrastructure and in places where there are public schools imply that one can expect some influence of private schools on public schools ability to attract students in areas with the stipend program. However, this should not be a major issue in this context given that private schooling is predominantly in the primary (and tertiary) sector.

<u>Third is the general welfare of the community- households residing in relatively poor</u> communities may respond differently to incentives than those from richer communities when everything else is constant. Controlling for these three types of initial conditions should help us better identify the impact estimates.

# _Regression discontinuity design_

The third way to deal with the selection bias is to use a regression-discontinuity design (RD design) technique that (interestingly) exploits the program design itself. RD design makes use of discontinuities generated by program eligibility criteria such that program assignment is based on a cut-off point of some assignment variable. In the Punjab case, 40 percent literacy is the cut-off point: all districts below this cut-off are eligible for girls stipends while those above are not. This is a case of _sharp RD design_<sup>_14_</sup> , where there is no region of common support in the propensityscore matching language of evaluation literature (Heckman et al. (1997), Rosenbaum and Rubin (1983)). In other words, probability of being exposed to treatment changes from 0 to 1 discontinuously as one crosses the cut-off (see Figure 1). RD design identifies and estimates the program impact in the neighborhood of the cut-off point for selection into the program. Within a small interval of the cut-off, schools on either side of the cut-off have similar values of district literacy rate and can be thought to have similar enrollment growth in absence of stipends or in presence of stipends, and therefore form “very good” comparison groups<sup>15</sup> . Since stipend eligibility is determined such that S=1 for literacy (L)<40% and S=0 L>40%, for some arbitrarily small value h>0, the impact estimate is given by

> 14 The other case is called fuzzy RD design where a cut-off of some variable does not fully determine one’s treatment exposure.

> 15 A more formal treatment of RD designs is found in Hahn et al. (2001).

12
In the RD literature, there are two broad methods to calculate this net impact: non-parametric and parametric. While non-parametric approach avoids any misspecifications in that of the parametric methods, it effectively uses a smaller sample of observations close to the cut-off point. Parametric specification on the other hand extrapolates from above and below the cut-off point to all observations in our data, and also allows for heterogeneous impact estimates. We will use parametric approach for most of our analysis but we also present results from non-parametric methods in our sensitivity checks.

# _Parametric approach of RD design_

Following Van der Klaauw (2002), our parametric specification is the following:

> Where λ ( _L_ ) is a control function of district literacy rate L. Since L is the only systematic determinant of stipend status (S) in our _sharp RD design_ , L will capture any correlation between S and the error term in equation (1), thereby freeing S from any possible endogenous

> contamination. α , therefore, provides a consistent estimate of the program impact. To allow for heterogeneous impacts across literacy spectrum and non-linear specification of the control function, one can also estimate the following regression specification that includes quadratic<sup>16</sup> of the control function and their interactions with S:

Where _L_ 40<sup>denotes the cut-off value, 40% literacy rate, for stipend eligibility. In this framework,</sup> α still gives us the average impact effect at the cut-off, but now one can also see differential average impacts across various program districts. From a policy point of view, impact effect at the cut-off may be more relevant. If one wants to expand the program to districts above the cut-

> 16 We try all specifications in our analysis: linear (only linear terms of main effect and interaction effect), quadratic (linear and quadratic) and cubic (linear, quadratic and cubic) terms of main effects and interaction effects. Use of ( _L_ 40 − _L_ ) term in the regression allows us to conveniently read off the coefficient on stipend dummy and interpret it as the impact estimate at the cut-off value.

13
off, this “marginal” impact estimate will give a picture of what the response may be from those areas.

Our DDD estimator can also be estimated within the RD design by adding a control function of literacy rate in our DDD regression specification:

# _Non-parametric approach of RD design_

Non-parametric RD method relaxes those restrictions in parametric approach such as model specification for outcomes and its functional forms. We implement this less restrictive nonparametric approach using local linear regression (LLR)<sup>17</sup> as suggested by Hahn et al. (2001) and Porter (2003) and used in Ludwig and Miller (2005). The LLR can be estimated using the following equation: (7) weighted local linear regression, where data points close to the cut-off have bigger weights and those further away have smaller weights. α obtained from this non-parametric approach provides us with the average impact effect at the cut-off. Standard errors of this local average treatment effect are calculated using bootstrapping method. We also estimate α _DDD_ using this strategy by including female dummy and interaction term of female and stipend status to equation (7).

> 17 Alternatively, one can use standard kernel estimators (such as those from one-sided kernel regressions) but these can suffer from possible boundary problems and bias in impact estimates (Hahn et al 2001).

> 18 We use Epanechnikov kernel: K(Z)=(3/4)(1-Zsquared) for |z|<=1, 0 otherwise. This kernel is widely thought to be most optimal for local linear regressions.

> 19 Choice of bandwidth is somewhat ad-hoc since there is no widely agreed upon method for optimal choice. We use a range of choices: 4, 8 and 12 for our sensitivity analysis. Larger bandwidth would increase bias while smaller bandwidth increases variance. Choice of bandwidths 4, 8 and 12 give positive kernel weights to approximately 1600, 2600 and 3600 observations out of 5000 sample size of girl middle schools.

14
# **4. Data**

Having discussed the estimation strategy in the previous section, we briefly describe the data before moving onto the results of our impact analysis. We use school-level census data from Punjab EMIS for 2003-2005. Even though school census data exists for all three years (2003, 2004 and 2005), majority of our analysis is based on data from 2003 and 2005 censuses. The main reason for doing this is the following: despite the fact that stipend program started at the beginning of 2004, the usual lag in both implementation (supply side) and response from the intended recipients (demand side) means that the full meaningful effect could not have taken place until a year later, thereby making 2005 a more suitable post-program data point.

Table 2 presents aggregate enrollments in grades 5 through 9 by year, gender and stipend eligibility of districts. For any given year and grade, girl enrollments in public schools are much lower than boy enrollments in each of stipend and non-stipend districts as a group. Gender bias is more pronounced within low literacy stipend-districts. In terms of trends, growth between 2003 and 2004 is prominent only for grade 6 girl enrollments in stipend districts. For grades 7 and 8, marked changes in girl enrollments are seen only by 2005, suggesting a lag in program impact, justifying our choice of 2005 as the end year of our impact analysis. Enrollment growth among boys are also large in both treatment and control districts, implying there are other forces in effect (not to mention other common reform components) in both areas since stipend are targeted to girls only. Grades 5 and 9 also record notable growth, but more so among girls and in stipend districts suggesting both “pull” and “push” effects of the program. Figures 2 and 3 show levels and growth rates, respectively, for combined enrollments for grades 6-8 by gender and stipend status.

EMIS data also has some useful school-level covariates such as infrastructure variables: rural/urban location, availability of water, electricity, toilet, and boundary walls. We generate yes/no (1,0) categories of each of these variables. We also calculate student-teacher ratio at the school level as the ratio of total enrollments to total number of working teachers. The extent of private schooling is proxied by ratio of private enrollments to public enrollment in a given locality. Data limitation makes us use private enrollment data from 2000 Census of Private Educational Institutions in Pakistan (PEIP 2000) and public enrollment from EMIS 2003. Ratios are calculated at the district level and provide an indicator for initial share of private sector that might affect growth in public enrollment later in time. Economic welfare indicator is measured by 15
average poverty rate at the district level as estimated from Pakistan Integrated Household Survey (PIHS) 2001-02. Finally, literacy rate - the deterministic variable for program eligibility- comes from the 1998 Population census.

We construct two main outcome variables: 1) absolute change in enrollment in grade 6-8 between 2003 and 2005 and 2) percent change in enrollment in grade 6-8 between the two periods. For secondary program effects, we also analyze enrollments in grade 5 and in grade 9 separately to gauze any possible _pull_ and _push_ effects. As mentioned earlier, schools are our primary units of analysis. For a balanced comparison, we restrict our impact analysis to those schools that have non-zero enrollments in each of grade 6, 7 and 8 for both pre-program and post-program periods.

Descriptive statistics of outcome variables and other covariates are presented in Table 3. Not only are there fewer girl schools for middle schooling level in stipend districts relative to both boys and relative to non-stipend districts, their average enrollments are lower than all other groups in both 2003 and 2005. But female middle schools in stipend districts (relative to three other groups) reported the largest increase in enrollments both in absolute level and percent terms. It should be noted that while the majority of schools saw a positive growth in enrollments, there were a sizable number of schools that reported decrease in enrollments in grades 6, 7 and 8. For instance, about 20% female middle schools and 25% of male middle schools in non-program districts recorded negative changes. On the other hand, only 7% of girl schools and 13% of boy schools in stipend districts recorded such negative growth.

In terms of other covariates, average literacy rates, not surprisingly, are lower and consumptionbased poverty rates are higher in stipend-districts. While the ratio of private schooling to public schooling is higher among non-stipend districts relative to stipend districts, it is higher for girls than for boys in both control and treatment districts. School infrastructure variables, in general, are more favorable in non-program districts. As one would expect, boundary walls and toilet facilities are available in higher proportion of girl schools. In terms of changes from 2003 to 2005, on average additional 8% of girl stipend schools built toilets; the comparable figures were 9% for female non-stipend schools, 19% for both types of male schools. And finally, studentteacher ratios are lower in stipend program districts, suggesting there is some space in their absorption capacity.

16
# **5. Impact Estimates**

We present the results of impact estimates obtained from different empirical strategies in tables 4- 11. Table 4 shows impact estimates of absolute growth in enrollments in middle school grades 6- 8 combined using basic double differencing (DD), DD combined with regression discontinuity design (RDD) and inclusion of other covariates. Table 5 does the same using triple differencing (DDD) and its combination with RDD and covariates. Tables 6 and 7 report impact estimates of percent growth using the specifications used in tables 4 and 5 respectively. Sensitivity analysis is presented in tables 8-10 and district level estimates in tables 11.

## _Absolute change in enrollments_

Average treatment effect of the stipend program is estimated to be an increase of approximately 12 students per school between 2003 and 2005 when using DD approach. Estimates change very little when we use regression-discontinuity design and add other covariates that are deemed to affect our estimates (table 4). Our preferred<sup>20</sup> specification is column (4) in the table: DD combined with RDD and other covariates produces very similar estimate. The last column (column 5) in the table controls for the initial size of enrollment in a school.

On the other hand, impact estimates from triple differencing (DDD) strategy are roughly half of those obtained from DD method. In this approach, the average program impact is an increase of 6 students per school (column 4 of table 5, the preferred model specification) and statistically significant at the 99% confidence level. Within DDD approach, results do not vary that much across different specifications.

If DDD strategy is taken to be more suitable than DD strategy in separating out the net impact of the stipend program for girls (as we argued in our methodology section), program impacts are overestimated using DD method even when we combine it with RDD approach and add other control variables.

While our main objective is to report impact estimates from the stipend program, results from our specification with other covariates (see columns 4 in both tables 4 and 5) provide useful information on whether the initial level of covariates under consideration have significant

> 20 Preferred because we believe it produces the best set of casual impact estimates after sufficiently accounting for selection bias and omitted variable bias.

17
association with change in enrollments. Lower the literacy rate higher is the change in enrollment (significant in DDD approach). Poverty rate, for instance, has a positive correlation. Share of private schooling too has a positive association. Infrastructure variables generally have positive correlations, with strongest effects for electricity and toilet facilities. Rural location seems to have a negative association.

# _Percent change in enrollments_

As reported in tables 6 and 7, size of impact estimates for percent change in enrollments vary across specifications within and between DD and DDD approaches. Basic DD estimate for percent change is over 23 percent (column 2, table 6), but it goes down to 10 percent when using RDD and covariates (column 4, table 6). Basic DDD estimate is 11 percent and that in our preferred specification (column 4, table 7) is 9 percent. Interestingly, the impact estimate obtained from DDD and covariates is statistically insignificant when initial level of enrollment is accounted for (column 5, table 7).

Illiteracy rate, poverty rate and share of private schooling all have positive associations with percent change in enrollments, while school infrastructure variables now have negative correlations. Interaction effects suggest that impact estimates on percent change in enrollments are stronger for rural areas but school-level infrastructure variables do not indicate differential effects.

# _Sensitivity analysis_

We present three sets of sensitivity analyses: i) applying our preferred specifications (RDD and covariates) to both DD and DDD for each of grades 6, 7 and 8 separately; ii) adding possibilities of heterogeneity impacts and non-linear functional form to our preferred RDD approach, and iii) using non-parametric local linear regression in RDD approach. Sensitivity of impact estimates are reported in tables 8-10.

While DD estimates for absolute increase in enrollments are significant for all grades, magnitude is the largest for grade 6 not surprisingly (table 8). But our DDD estimate for the same reports no impact for grade 6 but positive impact for grades 7 and 8. For percent change outcome indicator, impacts are seen only for grade 8 using DD and only for grades 7 and 8 using DDD.

18
As seen from table 9, impact estimates for absolute change do not vary significantly when using interaction of program status with literacy to account for heterogeneity and adding quadratic and cubic terms to account for non-linearity (column 2 and 3). Those for percent change, however, are sensitive to alternative specifications. For instance, none of the estimates using DDD are significant. Failure of some of these estimates to hold up with possible non-linearity and heterogeneity need not be concerning, it simply implies we do not need to introduce extra terms in our specifications.

Finally, use of non-parametric approach in RD design strategy produces interesting results. Close to the program eligibility cut-off point (e.g. using smaller bandwidth), impact estimates in terms of absolute changes are larger in magnitude (table 10). For instance, within 4 percentage points of the cut-off literacy rate (between 36 and 44), average impact estimate is an increase of 20 students for DD and 10 students for DDD estimators. Increasing the bandwidth to 12 percentage points on each side, estimates are reduced by half for each estimator. Estimates for percent change vary less across both DD and DDD and across choice of bandwidth, ranging from 16 to 21 for DD and 14 to 17 percent for DDD.

## _District level estimates_

Table 11 reports program impacts in terms of percent change in enrollments at the district level. Results, in general, support those obtained from our school level analysis for grades 6-8 individually and combined. While district-level analyses suffer from problems associated with grouped data, they provide an alternative check to our overall results. Moreover, the results for grades 5 and 9 indicate that DDD estimates are statistically insignificant implying absence of “pull” and “push” effects at early stages of the stipend program.

# **6. Supplementary Evidence from Repeated Cross-sectional Household Surveys**

We use child-level data from PIHS 2001-02 and PSLM 2004-05 household surveys to provide an additional set of estimates for the impact of girls-stipend program on school participation rate for 10-14 year old children. Because of limited sample size, the estimates of gross enrollment and net enrollment rates are not representative at the district-level (the surveys were meant to be representative up to the province level only). This does not allow us to do an in-depth impact analysis at the district-level. However, the aggregate impact estimates of the stipend program on school attendance that we provide in this section are statistically reliable assuming the sample 19
statistics are representative at least two groups of districts in Punjab: 15 stipend districts together as one treatment group and 19 non-stipend districts as one control group. Repeated crosssectional surveys, namely PIHS 2001-02 and PSLM 2004-05, are the only other independent sources of data available to us to compare our results from the provincial school census data.

The official age-group for middle school grades in Pakistan is 10-12 years. Age-grade profile from the survey data indicates that there are a large number of 13 and 14 year olds attending middle school grades because of either delayed entry to schools or higher repetition rates. For this reason, we focus on children of ages 10 to 14 years inclusive. Note that individual data from household surveys cover both public and private schools and the estimates on school attendance we present are for all types of schools. Descriptive statistics on the proportion of school attendance<sup>21</sup> for 10-14 year olds are shown in table 12. Levels and trends in these proportions follow similar patterns to those from the school census data. Girls in stipend districts have the lowest attendance rate in both periods, but they record the highest increase from 29 percent in 2002 to 48 percent in 2005. There is an increase of 7 percentage points for girls in non-stipend districts. Comparable figures for boys in stipend and boys in non-stipend are increases of 6 and 7 percentage points respectively.

We employ the same empirical techniques to estimate the impact of stipend program. Rather than schools as units of analysis, now the units of observations are individual children aged 10-14 years old. Because the surveys are repeated cross-sections and not pure panel at individual level, our regression specification includes interaction terms of stipend dummy and post-program time dummy in case of difference-in-difference (girls only) and a triple interaction term of stipend, post-program period and female dummy in case of triple differencing (girls _net_ of boys).

DD specification with RDD and covariates takes the following form:

> 21 One can also present results on an alternative indicator: proportion of out-of-school children. But as long as we measure progress in terms of percentage points change (and not percent change), results remain the same. For instance, increase in school participation rate from 29% in 2002 to 48% in 2005 implies proportion of out-of-school children decreased from 71% in 2002 to 52% in 2005. The change is 19 percentage points in both cases. In terms of percent change, one would be an increase of 66% while the other would be a decrease of 27%.

20
Where for each child i and time-period t, A is the probability of child’s student attendance (1=yes, 0=no), S is a stipend dummy (1=yes, 0=no), T is the post-program time dummy (1=yes, 0=no), ( _L_ 40 − _L_ ) is the stipend-eligibility cut-off minus the actual district literacy rate to reflect regression-discontinuity design, X is the set of child/household specific control variables, and α _DD_ the impact estimate.

Similarly, DDD specification with RDD and covariates takes the following form: 9) _Ait_ =μ+γ _Si_ +δ _Tt_ +ψ _Fi_ +ω _Fi_ * _Tt_ +ρ _Fi_ * _Si_ +κ _Si_ * _Tt_ +α _DDDSi_ * _Ft_ * _Tt_ +λ ( _L_ 40 − _Li_ ) +β _Xit_ + _eit_ In this framework, α _DDD_ , the coefficient on triple interaction of female dummy F, Stipend S and post-program period T, provides the impact estimate.

Impact estimates from basic DD and DDD specifications and those from DD and DDD combined with RDD and child-specific covariates are presented in table 13. These results use linear probability models. We also use probit models<sup>22</sup> and the results are similar to those from linear regression models.

Average treatment effect of the stipend program on the treated is estimated to be 12 percentage points increase in school attendance for 10-14 year old girls using the basic DD approach. When DD is combined with regression-discontinuity design and controls for individual characteristics, this impact estimate is 9.6 percentage points. In the case of DDD approach, average treatment effect is an increase of 13 percentage points in girls’ school participation rate _net_ of boys’ school participation rate. When RDD and covariates are added in this triple-differencing approach, the treatment effect remains high at 12 percentage points. All of these impact estimates are statistically significant at the 95% confidence level.

> 22 Compared to linear probability models, probit models are generally used for (0,1) categorical dependent variables such as the student attendance for three reasons: i) they ensure that predicted probabilities are between 0 and 1; ii) they allow and correct for heteroskedasticity present in (0,1) dependent variable; iii) they allow for heterogeneity in marginal effects of explanatory variable of interest. It should be noted that interaction effects and their standard errors in probit-type models from standard commands in statistical packages such as STATA (e.g., dprobit, mfx) are not correct. Results in our models are corrected using appropriate formulae for nonlinear combination of estimators (Wooldridge 2002).

21
Because stipends are given to all girls attending public middle school grades (despite the fact the target districts in the stipend program have the lowest literacy rates), it may be of general interest to look at equity implication of these conditional cash transfers. As seen in table 14, the school participation rates for these 10-14 year old girls vary significantly across per capita household consumption quintiles. More importantly, there was a large improvement in this indicator in stipend districts between 2002 and 2005, particularly for poorer quintiles. For instance, participation rate among 10-14 year old girls in the poorest quintile rose impressively from a 10% to 37% in that period. Regarding impact heterogeneity across welfare quintiles, the estimates indicate that the net program effects were stronger for the bottom two (poorest 40%) and the highest (richest 20%) quintiles, suggesting some kind of a U-shaped effect where stipends may have had a real effect on the poorer quintiles and the impact on the richest quintile is just picking up its disproportionate share and growth in private school attendance.

In terms of benefit incidence, we also see a noteworthy change from pre-stipend period to poststipend period. Aggregating all public middle school grade girl enrollments in the stipend program districts, PIHS 2001/02 and PSLM 2004/05 indicate that the current stipend transfers are neither pro-poor nor pro-rich. 50% of all stipends (and all other public resources in those public schools) go to girls from the poorest 50% of the population and the rest to the richest 50% (see table 15)<sup>23</sup> . Prior to the stipend program, the poorest 50% of girls were receiving only 38% of the government subsidy.

# **7. Conclusions**

This study evaluates the impact of a stipend program on female enrollments in government middle schools in the Punjab province of Pakistan. It makes use of a panel dataset of schools constructed from the provincial school census for 2003 (before the program) and 2005 (after the program). Because stipends for enrollees in grades 6-8 are targeted to girls in the 15 lowest literacy districts (out of 34) in the province, any simple naïve impact estimator (such as female enrollment grew by some y percent) is likely to suffer from a substantial program placement bias. The study considers estimates obtained using various empirical approaches including doubledifferencing (DD) and triple-differencing (DDD) in combination with regression-discontinuity design (RDD) and controlling for covariates. The results, not surprisingly, vary with

> 23 Sample size does not allow dis-aggregation by quintiles.

22
methodological choice. The preferred estimator derived from a combination of DDD and RDD empirical strategies suggests that the average program impact between 2003 and 2005 was an increase of 6 girl students per school in terms of absolute change and an increase of 9 percent in terms of relative change. These are modest but statistically significant program effects. We also make use of data from repeated cross-section household surveys (usually thought to be more independent and objective relative to administrative data such as the school census). Given that these household surveys are statistically representative only to the provincial level, any districtlevel analysis should be tempered with statistical caution. However, for what it is worth, we find an average treatment effect on proportion of school attendance for 10-14 year old girls ranging from 10 to 13 percentage points. Taken together, these results, strongly suggest that the stipend is attaining its objective of increasing female enrollment in public schools in Punjab. The evidence from the household survey data also points that the stipend program is helping children from poorer households to attend schools. Incidence of stipend subsidy also shows more equal distribution across rich-poor population groups.

Additional sources of data will help to test the robustness of our results (which essentially hinge on the veracity of the school census data), and more importantly, help address other pertinent issues besides enrollment increases. For example, Andrabi, Das, and Khwaja (2006) point out that the majority of Punjabi villages do not have middle/high schools. Sathar, Lloyd, Mete, and Haque (2000) have shown that access to school is one of the key determinants of enrollment (and other subsequent non-schooling outcomes such as contraceptive use) in rural Punjab. An immediate policy-relevant question then is what happens in villages in stipend districts which do not have a public middle/high school? To address this and other relevant questions we mentioned earlier, we are in the process of collecting relevant household and schooling data via primary surveys (instead of relying upon existing administrative data and limited household data).

23