metadata: authors: 0: author: Shapiro,Joseph S.
1: author: Skoufias,Emmanuel
count: Mexico
docna: 0: docna: Evaluating the impact of Mexico's
            quality schools program : the pitfalls of using
            nonexperimental data
docty: Policy Research Working Paper
subtopic: Educational Sciences,Educational Institutions & Facilities,Employment and Unemployment,Effective Schools and Teachers
teratopic: Education
abstracts: cdata!: The authors evaluate whether increasing
            school resources and decentralizing management decisions at
            the school level improves learning in a developing country.
            Mexico's Quality Schools Program (PEC), following many
            other countries and U.S. states, offers US$15,000 grants for
            public schools to implement five-year improvement plans that
            the school's staff and community design. Using a
            three-year panel of 74,700 schools, the authors estimate the
            impact of the PEC on dropout, repetition, and failure using
            two common nonexperimental methods-regression analysis and
            propensity score matching. The methods provide similar but
            nonidentical results. The preferred estimator,
            difference-in-differences with matching, reveals that
            participation in the PEC decreases dropout by 0.24
            percentage points, failure by 0.24 percentage points, and
            repetition by 0.31 percentage points-an economically small
            but statistically significant impact. The PEC lacks
            measurable impact on outcomes in indigenous schools. The
            results suggest that a combination of increased resources
            and local management can produce small improvements in
            school outcomes, though perhaps not in the most troubled
            school systems.
geo_reg_key: 119228,517191
repnme: repnme: Evaluating the impact of Mexico's
            quality schools program : the pitfalls of using
            nonexperimental data
display_title: Evaluating the impact of
            Mexico's quality schools program : the pitfalls of
            using nonexperimental data
listing_relative_url: /research/2006/10/7126808/evaluating-impact-mexicos-quality-schools-program-pitfalls-using-nonexperimental-data
topicv3: Teachers Management m1326119 1523,Education, Skills Development and Labor
            Market m1326135 1484,Education m1327048 470,School-Based Management m1327730 2143,Census Data m1504073 2763
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Education
no_of_pages: 48
author_keys: 0: author_key: 614684
1: author_key: 177660
model_extractions: WPS4036 <mark>I</mark> MPACT <mark>E</mark> VALUATION <mark>S</mark> ERIES <mark>N</mark> O <mark>. 8</mark>

**Evaluating the Impact of Mexico's Quality Schools Program: The Pitfalls of Using Nonexperimental Data**

## **Emmanuel Skoufias**

## **Joseph Shapiro**

The World Bank London School of Economics Email: eskoufias@worldbank.org Email: joseph.shapiro@gmail.com

# **Abstract**

We evaluate whether increasing school resources and decentralizing management decisions to the school level improve learning in a developing country. Mexico’s Quality Schools Program (PEC), following many other countries and U.S. states, offers US$15,000 grants for public schools to implement five-year improvement plans that the school’s staff and community design. Using a three-year panel of 74,700 schools, we estimate the impact of PEC on dropout, repetition, and failure using two common non-experimental methods: regression analysis and propensity score matching. The methods provide similar but nonidentical results. The preferred estimator, difference-in-differences with matching, reveals that participation in PEC decreases dropout by 0.24 percentage points, failure by 0.24 percentage points and repetition by 0.31 percentage points—an economically small but statistically significant impact. PEC lacks measurable impact on outcomes in indigenous schools. The results suggest that a combination of increased resources and local management can produce small improvements in school outcomes, though perhaps not in the most troubled school systems.

**JEL No** : I20, I21, I28, O22

**Keywords** : impact evaluation, school based management, matching, decentralization

# World Bank Policy Research Working Paper 4036, October 2006

_The Impact Evaluation Series has been established in recognition of the importance of impact evaluation studies for World Bank operations and for development in general. The series serves as a vehicle for the dissemination of findings of those studies. Papers in this series are part of the Bank’s Policy Research Working Paper Series. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ _Acknowledgments:_ For sharing data and providing useful comments, we thank Manuel Avila Carrazco, Ernesto Castellano Perez, Rodrigo Garcia Verdu, Paul Gertler, Edgar Ignacio Andrade Munoz, Alejandra Macias, Harry Patrinos and Marta Rubio.
# **Evaluating the Impact of Mexico's Quality Schools Program: The Pitfalls of Using Nonexperimental Data**

# **1. Introduction**

Recent economic research focuses on the quality of schooling, for good reasons. Evidence suggests that a one standard deviation improvement in math and science test scores can increase the real annual growth rate of per capita GDP by 1 percentage point (Hanushek and Kimko 2000, Barro 2001). Strengthening a person’s cognitive skills, as measured by standardized math, science, and reading tests, can increase that person’s earnings in adulthood (Murnane, Willett, and Levy 1995, Neal and Johnson 1996, Murnane et al. 2000, Altonji and Pierret 2001, Murnane et al. 2001, Lazear 2003). Moreover, improving the quality of schools can equalize income levels between racial and social groups (O’Neill 1990, Juhn, Murphy, and Pierce 1991 and 1993, Murphy and Welch 1992, Pierce and Welch 1996, and Hanushek 2004).

Quality of schooling does not merely refer to the quality of instruction in schools. Family income, a child’s home environment, initial and preschool education, non-school learning, and other factors significantly influence a person’s cognitive skills. Nonetheless, schools merit interest both since schools play an important role in building skills and since public policy can easily change schools.

Consensus remains elusive, however, as to what policy interventions can improve the quality of schooling. Experimental and non-experimental research has identified few educational inputs with statistically significant or economically large effects on learning (Glewwe and Kremer 2006). A large debate centered around Hanushek’s reviews (Hanushek 2003 provides the most recent) examined whether increasing school resources could improve learning, but recent reassessments by Krueger (2003) along with several natural and field experiments (Angrist and Lavy 1999, Kreuger 1999, Case and Deaton 1999, Chay et 1
al. 2005) have provided more convincing evidence that class size and perhaps other school inputs improve learning. One policy of particular interest involves decentralizing management decisions to the level of schools rather than national, state, or local bureaucrats. The discussion focuses not only on education—other policies have decentralized provision of public services with mixed results (World Bank 2004).

We examine the impact of a public program which sheds light on both school quality and decentralization debates—Mexico’s Quality School’s program ( _Programa Escuelas de Calidad_ , or PEC). We combine school census data, population census data, and program administrative data to evaluate the impact of PEC on dropout, repetition, and failure. PEC provides US$15,000 five-year grants to about ten percent of all Mexican public primary schools.

PEC started in 2001 with the goals of expanding autonomy and improving learning in Mexican preschools, primary schools, and secondary schools. Participation in PEC entails four activities. First, the staff and parents of a school prepare a plan which outlines steps for improving the school’s quality. Second, schools receive a five-year grant to implement the activities discussed in the school plan. In the first four years, PEC requires schools to spend 80 percent of the grant on supplies, infrastructure, and other physical goods. In the final year, schools must only spend 50 percent of the grant on such goods, and much of the grant funds teacher training and development. Third, PEC involves parent associations in designing school improvement plans, purchasing supplies, and carrying out the plans. Fourth, PEC trains school principals. Mexico requires no formal training of principals, and many principals switched from being teachers to principals without formal transition.

Every Mexican primary school may participate, but PEC targets disadvantaged urban schools through direct mail, radio, and other media. To identify disadvantaged schools, PEC uses a poverty index that the Oportunidades program and Mexico’s National Population Commission (CONAPO) constructed. To identify urban schools, PEC uses 2000 2
census data to select localities with more than 15,000 residents. In the 2001-2002 school year, 2,200 schools enrolled in PEC. But by the 2003-04 school year, 20,600 schools or 10 percent of all Mexican primary schools received PEC support.

We construct a panel of 74,700 schools and use two common non-experimental methods to create a control group and estimate impact: regression analysis and propensity score matching. We compare and contrast the estimated impact of PEC on dropout, repetition, and failure rates. The preferred estimator, difference-in-differences with matching, reveals that participation in PEC robustly decreases drop-out rates by 0.24 percentage points, failure rates by 0.24 percentage points and repetition rates by 0.31 percentage points.

The results provide useful information for a variety of other countries, such as El Salvador, Kenya, Kyrgyz Republic, Nepal, Nicaragua, Paraguay, and Yemen, that are developing or operating similar programs. They also guide countries seeking to improve learning and inform the broader debate on education decentralization.

The paper is structured as follows. Section 2 reviews research on decentralization and school-based management. Section 3 discusses data sources and variables used in the empirical analysis. Section 4 discusses the empirical approach, while section 5 estimates program impact. Section 6 presents results, and section 7 concludes.

# 2. Decentralization and school based management

Authors generally argue that decentralization – the transfer of financial, pedagogical, personnel, or other decision-making power from central to local authorities – may make policy better reflect beneficiaries’ heterogeneous preferences, and that local management increases the accountability of political authorities. Critics emphasize that centralization may decrease costs through economies of scale, and that decentralization may decrease the quality of public services if local providers lack technical capacity or if local elites 3
monopolize the benefits of public services (see, for example, Oates 1972. Besley and Coate 2003, and Galiani et al. 2004). Gunnarsson et al.’s (2004) model of education decentralization offers one reason why decentralization may improve outcomes: local decision-makers may know needs best. If principals, teachers, and parents know the areas that require spending and the technologies that improve learning better than national decision-makers do, then local authorities will spend on what is most needed. Eskeland and Filmer (2002) also theorize that school autonomy increases local power and that the participation of parents in schools pushes that power to be used for increasing student learning.

Data offer mixed support for these theories. Eskeland and Filmer (2002) find that, consistent with their model, the autonomy of teachers, principals, and parents to make organizational and pedagogical decisions and the participation of parents in schools significantly increase primary school test scores in Argentina. Galiani et al. (2004) find that Argentina’s decentralization of secondary schools significantly increased test scores overall but decreased scores for schools in poor areas and in provinces with pre-decentralization fiscal deficits.

A radical form of decentralization is school-based management, or the transfer of authority to principals, parents, teachers, and other actors in a school and its community. School-based management programs vary. Some programs give parents power to make decisions; others only give teachers and principals that power. Some transfer power to allocate budget; others also transfer power to hire and fire teachers, set curriculum, and change the school schedule. Some force schools to develop improvement plans; others do not. These programs also have different focuses. Some seek only to increase the freedom of school-level actors, some seek only to increase the participation of parents in a school, and some seek to increase the learning of students. The aspect of these programs which mainly distinguishes them from more common educational interventions – school building, teacher training, school feeding programs, or others – is their focus on allowing _school-level staff_ and 4
parents rather than district-, state- or national- level education staff to make management, curricular, or budgetary decisions. A limited group of other educational interventions might improve learning (see, _inter alia_ , Jalan and Glinskaya 2005; and Glewwe and Kremer 2006). We focus on examining whether the new breed of school-level management programs can improve learning.

Evidence on the impacts of these programs is mixed. A review of school-based management plans in the U.S. (Summers and Johnson 1996) finds four evaluations with comparison groups. Collins and Hanson (1991) compared mean outcomes in Dade County, Florida schools that the county had picked to create faculty councils with budgetary and personnel power against other schools without the councils. After three years, they found unchanged test scores but decreased dropout and suspension rates in schools with councils. Taylor and Bogotch (1992) mined the same data to find no significant correlation between teacher autonomy and student test scores. South (1991) compared trends in Scholastic Assessment Test (sat) test scores between 1985 and 1989 in Monroe County, FL, which gave schools power to make budget, personnel, and curriculum decisions, all of Florida, and all of the U.S. They found no evidence of better performance in Monroe County. Winfield and Hawkins (1993) studied a Philadelphia program which attempted to increase collaboration within elementary schools. They ran regressions including non-project schools as a comparison group, controlling for various background indicators. They associated the project with teachers purchasing basic materials and hardware but little change in reading test scores.

Jimenez and Sawada (1999) examine 1996 data on Community Managed Schools (EDUCO), a program created to expand coverage in rural El Salvador. With no background controls, they found that parents of students in EDUCO schools were three times more likely to engage in daily classroom activities and significantly more likely to meet with teachers. Overall, EDUCO’s poor students had lower mean Spanish and math exam scores than the 5
wealthier students of traditional schools did. When they control for the portion of EDUCO schools in each municipality, they find that EDUCO has no effect on mastery of mathematics but expanded mastery of language by about two subjects, or one standard deviation.<sup>1</sup> When they control for the factors that municipalities used to place EDUCO schools, they find that EDUCO has a positive but insignificant effect on language scores. They also found that EDUCO decreased student absenteeism. Sawada and Ragatz (forthcoming) use similar methodology to find that EDUCO has transferred few administrative processes to local levels but gives local actors greater perceived influence in hiring and firing teachers. Controlling for other factors, teachers in EDUCO spend more hours teaching, more often meet with parents, principals, and other teachers, are absent less, and attend more training sessions than teachers in similar non-EDUCO schools do.

King et al. (1999) evaluate School Autonomy, a Nicaraguan program from the mid1990s that selected public primary and secondary schools to create faculty or parent councils with power to change the curriculum, choose textbooks, independently evaluate students, hire and fire principals, and set monthly fees for students. In other public schools, the Ministry of Education oversaw these activities. They find that program participation significantly increases the portion of decisions made at the school level but did not impact the level of influence felt by principals or teachers.

# 3. Data Sources and Outcome and Control Variables

Our empirical analysis is based on a variety of data sources. These are the annual Mexico School Censuses, locality level data from the 2000 Mexico Population and Housing Census, and administrative data files from the PEC, Oportunidades and CONAFE programs. We combine all school data sources at the school level using unique school

> 1El Salvador’s math and language tests have 30 and 36 questions. Jimenez and Sawada define outcomes according to the number of subjects a student has mastered, where mastery is defined as correctly answering two

6
identifier codes, and then combine the school with the National Census data sources using unique locality identifier codes.

To identify PEC schools, we use administrative data on PEC coverage in 2001, 2002, and 2003.<sup>2</sup> One key variable for our analysis is the definition of the treatment variable. Specifically how should one define coverage by PEC? The construction of this variable plays a critical role if one seeks to evaluate the impact of PEC. We choose to work with two alternative definitions of what constitutes participation in PEC. The first treatment variable (T) is based on a strict criterion, requiring that a school must have received PEC funds in all three school years covered by the school census data we were able to access. Thus, T=1 identifies schools that received a PEC grant in school years 2001, 2002, and 2003 is classified in the treatment group, whereas T=0 identifies schools that did not receive a PEC grant in the same school years (the control group).<sup>3</sup> Based on the strict definition of treatment, there are 1,767 schools in the treatment group and 65, 457 in the control group, each school observed in 2000 and in 2003.

Given that the number of schools covered by PEC increased significantly in 2002 and 2003 we also constructed an alternative treatment variable (T2) based on a less strict criterion. The treatment variable T2=1 identifies schools that received a PEC grant in any of the three school years, while T2=0 identifies the schools that did not did not receive a PEC grant in any of three school years (the control group).<sup>4</sup> Based on the less strict definition of treatment, there are 9,244 schools in the treatment group and 65, 457 in the control group, each school observed in 2000 and in 2003.

The main outcome variables we examine are the average (across all six grades) dropout, failure and repetition rates in the school. To measure dropout, failure, and

> of three questions (math) or three of four questions (language) in a subject.

> 2We identify a school year by the calendar year in which it began, so 2001 refers to the 2001-2002 school year.

> 3 Thus schools that received a PEC grant in one or two of the three school years since the start of PEC in 2001 are excluded from the analysis when the variable T is used as a measure of treatment.

7
repetition, we use Mexico’s School Census (also called Statistics 911), an annual listing of background and outcome data for Mexican primary schools. The dropout rate in any given school year t (spanning calendar years t and t+1) is defined as 1- (number of students enrolled at end of school year t divided by the number who enrolled at any time in school year t). The failure rate in school year t is defined as 1-(number of students who passed grade in school year t divided by the number who were enrolled at end of school year t). Lastly, the repetition rate in school year t is defined as 1-(number of students who were repeating their grade at beginning of school year t+1 divided by total enrollment at beginning of school year t+1). Mexico’s school census also includes some information on the schools themselves: the number of classes, the ratio of teachers to students, the school type (indigenous or non-indigenous), and the number of rooms in the school. We include each of these variables, as a school’s size and teaching burden may influence its likelihood of requesting PEC funds.

To measure locality background data, we use a version of Mexico’s 2000 Census of Population and Housing prepared by the Mexico’s National Population Council (CONAPO). CONAPO used the complete Census data to construct a poverty index for each Mexican locality. PEC advertises to schools based on the CONAPO marginality index of a school’s locality. That index is a function of eight variables from Mexico’s 2000 census. Rather than use the index itself, we use the richer information contained in these eight locality-level variables. Mexico conducted the census only in 2000, giving data on these outcomes only for the baseline of PEC.

Lastly, we include participation data from Mexico’s National Council for Educational Development (CONAFE) and Oportunidades (formerly PROGRESA), two social programs run by Mexico’s federal government. CONAFE targets mainly rural areas while Oportunidades has expanded from a rural focus to offer scholarships to poor urban students 4 Thus the control groups for both definitions of treatment are identical.

8
as well. We include a dichotomous variable as to whether the school received CONAFE compensatory program funding in 2001 and a continuous variable indicating the portion of students in the school in 2001 that received an Oportunidades scholarship.

Most training evaluations include pre-program earnings and pre-program employment trends. Heckman et al. (1999) emphasize that the variables to include as controls are those that determine participation and are not determined by participation into the program. Some emphasize that the similarity of pre-program outcomes between comparison and treatment groups can be a litmus test for the acceptability of comparison groups in evaluating training programs (Heckman and Hotz 1989, Angrist and Krueger 1999). By definition, PEC participation cannot affect pre-program outcomes or trends, but these variables may influence a school’s decision to participate in PEC. Following such work, we include one-year pre-intervention one-year trends in matching and regression controls. Longer-term pre-intervention trends might better capture factors which affect the trend of education outcomes in the school. Since our main estimating equations use differences-in-differences, which eliminate time-invariant school and community factors, we have more concern in capturing short-term changes in schools – changes in principals or teachers, an influx of migrants, new facilities, or others – which would affect outcomes during the PEC intervention.

_Changes in intermediate outcomes_ Before describing method and results of our estimate of PEC’s impact on outcomes, we discuss a reflexive comparison of changes in intermediate outcomes in PEC schools using data on changes between June 2002 and June 2004 from surveys of students in 505 PEC schools. This information serves two purposes. First, by showing trends, these results show if PEC schools are moving towards the desirable goal of having schools function more effectively. Second, by providing detailed information on the state and trends of parent9
school relations, student satisfaction, and teacher performance, these data suggest areas on which any program to improve school quality in Mexico might focus. Since education in all Mexican schools was changing during this time and these data are not available for non-PEC schools, these results do not allow attribution of changes to PEC. Nonetheless, they suggest pathways by which PEC might affect any outcome.

Overall, students in PEC schools report improved school infrastructure and security, unchanged involvement of school principals, increased parental participation in schools and in students’ homework, some improved and some unchanged teaching practices, and increased expectation by parents and students that students would complete advanced education. Despite these relatively rosy self-reported results, both reading and math test scores decreased by statistically insignificant amounts.

Students reported statistically significant improvements in school infrastructure and security. Compared to 2002, in 2004 a greater portion of students reported that school spaces had recently improved and that everything in the school was very organized. The portion of students who claimed to feel safe and secure from danger increased by three percentage points during those two years. Still, a quarter of students reported that they did not feel safe from danger while at school.

While 8 of 10 principals spoke with students and 9 of 10 principals visited classes, students reported small and statistically insignificant decreases in principals visiting classes or speaking with students. Students also reported improved parental and tutor support for and involvement in education at home. Students indicated that parents became 5 percentage points more likely to help with students’ homework, less likely to interrupt when students were doing homework, more likely to help students study for exams, more likely to read a student’s textbook, and more likely to explain what students did not understand from class. Nineteen of 20 parents spoke with teachers, but parents became more likely to help in 10
activities that teachers or principals request and more likely to worry about whether school was going well.

Teachers exhibited smaller changes. About half of teachers arrive to class late - a proportion that remained unchanged - but 99.4 percent of teachers attend class every day. Teachers were reported to be “more happy”, to have more patience, to yell and get angry less often, and to less often speak with other teachers during class. Teachers also became significantly more likely to encourage students to continue studying. Several teaching practices - using teamwork, reviewing material that students did not understand, and commenting on homework - remained frequent. Students reported few changes in the efforts of teachers to involve parents in schools.

Students reported no changes in their reading practices at home, and that their satisfaction with school, their class, and their teacher remained high and unchanged. Students did report that they were more likely to hope to continue studying past primary school.

# _Descriptive statistics for PEC schools_

Table 2 describes the variables, and Table 3 presents mean values for these covariates. Compared to non-PEC schools, PEC schools are less likely to be in an indigenous locality and likely to have more students per teacher, more rooms, and more classes. PEC schools are significantly more likely to be in urban areas and are at higher altitudes. Compared to the localities in which non-PEC schools are located, the localities of PEC schools have lower illiteracy rates, lower levels of educational attainment among adults, but better access to sanitation, electricity, and water. PEC localities have a greater portion of adults who earn less than twice the minimum wage. Overall, this description fits the portrayal of PEC as generally serving urban areas (with better infrastructure than rural localities) but poor localities.

11
# 4. Empirical Approaches to Estimating PEC’s Impact

We seek to identify the impact of PEC: what outcomes would PEC students have had if PEC had not existed? Given that we have non-experimental data, we cannot distinguish the effect of treatment from the bias generated by a nonexperimental estimator (Smith and Todd 2001). Rather than rely on one non-experimental method with its strengths and limitations, we choose two and compare the impact estimates that we obtain by each method. The first method is based on linear regression analysis and the second on propensity score matching. Comparing the impact estimates that we obtain by each method highlights the weaknesses of the simpler methods and probes all methods for consistency in results.

The central problem in the evaluation of any program is the fact that individuals participating in the program cannot be simultaneously observed in the alternative state of no treatment. To illustrate, let<sup>_Y_</sup> 1<sup>be the outcome for a given student in the treated state (i.e.,</sup> during her school’s participation in the PEC program) and _Y_ 0 is the outcome in the untreated state (i.e., a non-PEC school). Then the gain for any given individual or household from being treated by the program is Δ = ( _Y_ 1 − _Y_ 0 ) . However, at any time a person is either in the treated state, in which case<sup>_Y_</sup> 1<sup>is observed and</sup> _Y_ 0 is not observed, or in the untreated state, in which case<sup>_Y_</sup> 1<sup>is unobserved and</sup> _Y_ 0 is observed. Given that missing<sup>_Y_</sup> 1<sup>or</sup> _Y_ 0 preclude measurement of this gain for any given individual, one has to resort to statistical methods as a means of addressing this problem (e.g., see Heckman, LaLonde and Smith, 1999). The statistical approach to this problem replaces the missing data on persons using group means or other group statistics, such as medians.

12
For example, the majority of the studies on evaluation of social programs focus on the question of whether the program changes the mean value of an outcome variable among participants compared to what they would have experienced if they had not participated. The answer to this question is summarized by one parameter called the “Average effect of Treatment on the Treated” (ATT). Using formal notation, the ATT effect (denoted by the expectation operator _E_ ) of treatment on the treated (denoted by _T_ =1) with characteristics _X_ may be expressed as:

> The term _E_ ( _Y_ 1 | _T_ = ,1 _X_ ) can be reliably estimated from the experience of program participants. What is missing is the mean counterfactual term _E_ ( _Y_ 0 | _T_ = ,1 _X_ ) that summarizes what participants would have experienced had they not participated in the program.

The variety of solutions to the evaluation problem differ in the method and data used to construct the mean counterfactual term _E_ ( _Y_ 0 | _T_ = ,1 _X_ ) . Generally, the preferred approach is that of social experimentation or randomization of individuals (or schools) into treatment and control groups. Experimental designs use information from individuals in the control group to construct an estimate of what participants would have experienced had they not participated in the program, i.e., the term _E_ ( _Y_ 0 | _T_ = ,1 _X_ ) . Random assignment into treatment and control groups equalizes the mean selection bias between the treatment and control groups, which then is eliminated when one considers the differences in

_E_ ( _Y_ 1 | _T_ = ,1 _X_ ) and _E_ ( _Y_ 0 | _T_ = ,1 _X_ ) . In the absence of an experimental design, one has to resort to alternative methods that involve behavioral assumptions which are typically difficult to test and more or less frequently violated.

13
The most common problem in the evaluation of a program with non-experimental data concerns the bias arising from self-selection. In practice, it may be impossible to observe and quantify all the variables that are critical in determining a school’s participation into the PEC program. An example of such a variable would be the motivation and drive of the school principal. In practice, the extent to which selection into a program is based on unobservable variables is something that cannot be determined either ex-ante or ex-post. This fact, in turn, explains the preference towards randomized designs which ensure the equalization of biases between control and treatment groups no matter how selection into the program takes place in reality. In the absence of experimental data, the common practice is to employ an assumption regarding the determinants of participation into a program. For example, selection into the PEC program is based on variables that are observables. This in turn implies that selection bias can be eliminated by conditioning on these observable variables.

Given the data at our disposal, this is the approach that we are forced to adopt.<sup>5</sup> However, we are able to do better than that. The panel nature of our data allows us to permit selection into the PEC program to be based on observables as well as unobservable variables which are time invariant. If (a) the motivation and drive of school principals is constant over time, or (b) this motivation changes only due to PEC participation, or (c) the changes of motivation not due to PEC participation do not correlate with PEC participation, then we can be reasonably confident that our estimate of program impact is free of any selection bias arising from either observed or unobserved variables.

In the remainder of this section we discuss the two alternative methods we employ to obtain estimates of the impact of PEC on key outcomes.

14
_Estimating PEC’s impact using Regression Analysis_ We form a panel of schools observed in 2000 (baseline) and in 2003 (post-intervention year) and estimate a linear regression of the form where _Y(s,t)_ denotes the value of the outcome indicator in school _s_ in period _t_ , α , β , and γ are fixed parameters to be estimated, _T(s)_ is a binary variable taking the value of 1 if the school participates PEC (i.e., is in the treatment group) and 0 otherwise (i.e., belong in the control group), _y03_ is a binary variable equal to 1 for school observations from the 2003-04s school year (after the initiation of the program) and equal to 0 for the 2000-01 (the year before the initiation of the PEC program), **_X_** is a vector of school and village characteristics, and _U_ ( _s_ , _t_ ) is an error term summarizing the influence of unobserved disturbances. It is assumed that the disturbance term _U_ ( _s_ , _t_ ) has two components, as in where μ( _s_ ) represents an unobserved effect that does not vary over time but does vary between schools, and ε( _s_ , _t_ ) represents an effect which varies both over time and across individuals. If μ( _s_ ) or ε( _s_ , _t_ ) are correlated with the PEC participation (treatment) variable _T_ , then the estimate of PEC impact captured by the parameters β _T_ and β _Ty_ is likely to be biased. In our empirical analysis we allow μ( _s_ ) to be correlated with _T_ , but we assume that ε( _s_ , _t_ ) is a pure disturbance term that is uncorrelated with _T._<sup>_6_</sup> In order to provide the reader with a better understanding of the specification in eq.

(2) it is best to divide the parameters into two groups: one group summarizing differences in

> 5 There is also the alternative of trying to locate an instrumental variable (or more) that can be used to control for the role of selection bias arising form selection based on unobservables. As Heckman (1997) argues, proper instrumental variables are very difficult to find.

> 6 All of our regression-based estimates are based on “robust” standard errors which means that we also allow for heteroskedasticity in ε( _s_ , _t_ ) .

15
the conditional mean of the outcome indicator before the start of the program (i.e., α , and β _T_ ,) and another group summarizing differences after the start of the program (i.e., β _y_ , and β _Ty )_ . Specifically, the coefficient β _T_ allows the conditional mean of the outcome indicator to differ between schools in treatment and control before the initiation of the PEC program, whereas the rest of the parameters allow the passage of time to have a different effect on households in treatment and control schools. For example, the combination of parameters β _y_ and β _Ty_ allow the passing of time (after the start of the program) to affect PEC schools differently from non-PEC schools.

Based on the preceding specification, the conditional mean values of the outcome indicator for treatment and control groups before and after the start of the program are as follows: Based on these conditional means one can then easily derive the three different estimates of program impact that are common throughout the program evaluation literature: the cross-sectional difference estimator, the before and after difference estimator and the difference-in-differences estimator. These three alternative estimators of impact are discussed next.

The **cross-sectional difference (CS)** estimator of impact in 2003 is given by the expression<sup>7</sup> : 16
Expression (8) highlights the fact that the estimated impact of the program may be biased by any pre-program differences between treatment and control groups (summarized by the β _T_ term) and the mean difference in time invariant unobservables between the treatment and the control group (i.e. [ _E_ (μ( _s_ ) | _T_ = 1 ) − _E_ (μ( _s_ ) | _T_ = 0 )] ).<sup>8</sup> The presence of that latter term, in particular, suggests that unobservable variables may not only affect the cross-sectional estimate of the ATT effect directly, but also indirectly through the bias they might impart on the other parameters of the model such as β _T_ and β _Ty_ . In principle, a credible estimator of

program impact should be free of any biases inherited by pre-existing (pre-program) differences between the treatment and control groups as well as from differences in the mean values of the unobserved factors that may be correlated with the decision to participate in PEC.<sup>9</sup> The **before-and-after difference (BA)** or **reflexive** estimator is given by As expression (9) reveals, the program impact estimated by the BA estimator includes the trend or aggregate effects in the changes of the outcome indicator Y (summarized by the β _y_ term). Thus, the BA estimator may attribute a large impact to the PEC program, when in fact most of the change in the outcome variable _Y_ would have taken place anyway even without the presence of the PEC program.

> 8 Note that in an ideal randomized design, [ _E_ (μ( _s_ ) | _T_ = 1 ) − _E_ (μ( _s_ ) | _T_ = 0 )] =0. 9 An alternative to the specification of eq. (2) is to replace the vector X by a set of school-specific binary variables or fixed effects. This alternative specification allows us to estimate impact only through DID. We have also estimated equation (2) using fixed-effects in place of the X-vector of school and locality characteristics and confirmed that the DID estimate is identical to the one that is presented herein.

17
The widely preferred estimator of program impact in this context is the **differencein-differences estimator (DID)** . Assuming that the aggregate effect, summarized by the estimate of β _y_ , is identical for both treatment and control groups, one may obtain an estimate of the impact of a program that is free of any of the biases that are likely to contaminate the BA and CS estimators. For example, the DID may be viewed as removing the undesirable components from the CS estimate of program impact in 2003 ( β _T_ and [ _E_ (μ( _s_ ) | _T_ = 1 ) − _E_ (μ( _s_ ) | _T_ = 0 )] in eq. 9) by subtracting from it the cross-sectional differences between the same groups in 2000. Along similar lines, the DID may be thought of as removing the aggregate or trend effect β _y_ from the BA estimate of program impact described in equation (9). Using the specification of equation (2), the DID estimate of the program impact can be summarized by the single parameter β _Ty_ , i.e.

DID estimates using OLS with many years of data may suffer from severe serial correlation, as Bertrand et al. (2004) demonstrate using placebo laws on U.S. states. The use of few time periods with many groups partly addresses the concern. Furthermore, we focus on discussing nonparametric propensity score matching over the OLS results due to this and other aforementioned reasons.

Mexico’s education data generally suggest that trends in education outcomes are non-negligible; hence, a before-after estimator is unlikely to reflect PEC’s true impact. Since PEC is a voluntary program, PEC schools likely differ from non-PEC schools in factors that will affect both levels and trends in outcomes. Hence, conventional cross section or before and after comparisons are unlikely to capture PEC’s true effect.

Even though the DID estimate of program impact is the preferred estimator for the reasons outlined above, we also present and discuss briefly the other impact estimators as a 18
means of getting a better sense of the extent to which there are biases in cross-sectional and before-and-after estimates of impact.

_Estimating PEC’s impact using Propensity Score Matching_ Propensity Score Matching (PSM), first proposed by Rosenbaum and Rubin (1983), provides an alternative, and, increasingly preferred non-experimental approach for evaluating program impact. Unlike regressions, matching has the advantage that it does not require an analyst to assume linear relations between treatment, covariates, and outcomes. Matching’s agnostic nonparametric method may result in more accurate estimate of impact. Also, in calculating the expected counterfactual for each treated observation, matching weights the observations differently than an ordinary least squares regression. In the regression-based estimates of program discussed above all the non-PEC (untreated) schools have a role in determining the expected counterfactual for any school receiving PEC (treated schools). In contrast, in the PSM method the comparison of outcomes is performed using PEC and nonPEC schools that are as similar to each other as possible.

As in the regression approach to impact evaluation, a key assumption for the validity of the matching method is that selection into the program is based on observable variables. The method proposes to summarize pre-treatment characteristics of each subject into a single index variable (the propensity score) which makes matching feasible. The propensity score is defined as the conditional probability of receiving a treatment (participating in PEC) given pre-treatment characteristics _X, i.e._ where _T_ is the indicator for exposure to treatment (=1 if in PEC, =0 if non-PEC). As shown by Rosenbaum and Rubin (1983), if the propensity score _p_ ( _X s_ ) is known, then the ATT can be estimated as follows 19
where the outer expectation is over the distribution of ( _p_ ( _X s_ ) | _Ts_ = 1 ) , and the subscript s indexes the specific school.

Following the common practice, we use a probit model to estimate the propensity score. The list of variables included in the probit model (the elements of the vector X) is practically identical to the list of variables used in the regression model (2). We then apply _local linear matching_ which is analogous to running a weighted regression for each PEC school on only a constant term using all the non-PEC school data.<sup>10</sup> We also verify balance in mean values of covariates between the treated and matched comparison groups. To measure confidence in this estimate, we repeat the matching process 50 times, each time drawing a random sample with replacement that results in the same number of observations as in our dataset. Since we sample with replacement, some observations are sampled twice and some never. The process mimics the data collection process and produces 50 estimates of average effect of treatment on the treated (ATT). We then estimate a standard error for these 50 estimates. While bootstrapping standard errors for matching estimates is common, no formal justification of it has been provided and a debate has recently arisen as to the acceptability of bootstrapping in this context (Abadie and Imbens 2005). However, since it provides the most direct way to measure error in estimating propensity scores and in matching observations, we measure error by bootstrapping.

Since PEC schools may differ significantly from non-PEC schools, the inclusion of comparison observations outside the common support in the regression may skew estimates of PEC impact. Propensity score matching controls for many observable differences between PEC and non-PEC schools, but ignores all the unobservable factors involved in the decision to participate in PEC. In particular, since we have data on few school-level factors, it is

> 10 Local linear matching converges faster at boundary points and can adapt better to different data densities.

20
unlikely that we can satisfy the assumption of conditional independence–the assumption that, given observable factors, PEC participation is unrelated to a school’s potential outcomes. Differences-in-differences matching may offer the estimate that is closest to PEC’s true impact. This estimator eliminates time-invariant unobservables. It also assumes that covariates do not change over time between PEC and non-PEC schools, and that the effect of covariates does not change over time.

Given the availability of panel data on outcomes, we also use the differences-indifferences matching (DIDPSM) estimator developed in Heckman et al. (1997, 1998a, 1998b). This estimator compares the before-after drop-out, failure and repetition rates of PEC schools with the corresponding before and after changes among non-PEC schools, conditional on covariates _X_ . This builds on the simple differences-in-difference statistic by controlling for covariates _X_ and estimating the differences using semiparametric methods: where the subscript _t_ denotes post-treatment observations, and _t_<sup>_’_</sup> denotes pre-treatment observations. The advantages of this estimator are similar to the advantages of the DID regression – it eliminates any unobserved factors that vary between observations but not over time. As is I sthe case for the DID estimator using regression analysis, equation (13) assumes that given covariates _X_ , the trends for PEC and non-PEC schools would have been identical in the absence of PEC: Thus, difference-in-differences matching allows selection on unobservables as long as unobservable factors do not vary between observations and over time. Taking into consideration the advantages over the DID based on regression analysis, the DID with matching is our preferred estimator of program impact.

21
# 6. Results

## _Learning and Enrollment Outcomes_

We begin with a simple comparison of the mean drop-out rates, failure rates and repetition rates using the two alternative definitions of treatment or coverage by PEC, and without controlling for any background factors (see table 4). Put in another way, these preliminary estimates do not make any effort to control for the fact that selection into the program is voluntary. Such a comparison of the key outcome variables would be more appropriate if participation or no-participation into PEC was determined through a national lottery, where each school had an equal chance of participating into PEC. Given that in reality, participation into PEC is determined on a voluntary basis, it is important to keep in mind that such a comparison is more useful for seeing differences and trends between PEC and non-PEC schools rather than for determining the causality of outcomes.

The CS estimates show that PEC schools have half a percentage point higher dropout but two percentage points lower failure and repetition rates than non-PEC schools. The BA estimates, on the other hand, show that between 2000 and 2003, the mean dropout rate in PEC’s schools decreased by 0.41 percentage points and mean failure and repetition rates decreased by 0.87 and 1.06 percentage points.

A simple differences-in-differences comparison between PEC and non-PEC schools shows that dropout rates decreased by 0.20 percentage points faster in PEC schools while failure rates decreased by 0.18 percentage points faster in non-PEC schools. Since these estimates do not control for any background factors, they may not reflect PEC’s true impact.

In table 5 we present estimates of program impact by taking into account the role of observable characteristics likely to influence participation into the program.<sup>11</sup> Our discussion focuses on comparing and contrasting the various estimates of program impact. We first 22
compare CS, and BA, against the DID using regression analysis. Then, we compare DID using regression vs. DID using PSM.

We begin with a discussion of the estimates of the average treatment of the treated effects (ATT) obtained from the regression model of equation (2), as these allow us to compare the impact obtained by the CS, BA and DID estimators. As suspected, the CS and BA estimates of program impact are quite misleading. The BA estimate of the impact of PEC on the drop-out rate (row 4) suggests that the program led to a decline of 0.411 percentage points. However, the BA estimate of the trend obtained for non-PEC schools (row 5) suggests that more than half of this decline would have taken place anyway in the absence of PEC. A similar pattern holds for the BA estimate of PEC’s impact on failure and repetition rates. The BA estimates suggest that among PEC schools failure declined by 0.867 percentage points (row 11) and repetition by 1.064 percentage points (row 14).

We mentioned earlier, the CS estimates of program impact are likely to be subject to even more biases. In 2003 the CS estimate of PEC’s impact (row 2) suggests that there are no significant differences in the drop-out rates between PEC and non PEC schools in 2003 (row 2) while the failure and repetition rates in PEC schools are significantly higher in PEC schools. These cross sectional differences do not take into account the differences in drop out rates that prevailed in 2000, prior to the start of PEC. In 2000, for example, drop-out rates were higher in PEC schools than non-PEC schools (row 1). Combined together these crosssectional differences in 2000 and 2003, suggest that these differences between PEC and nonPEC schools have decreased over time which implies that the program may be reducing drop–out rates, as well as failure and repetition rates.

The preferred estimates, the DID, estimate impact by differencing the cross –sectional difference estimates of impacts in 2003 and in 2000, or by differencing the BA estimates of

> 11 The full set of estimates is contained in appendix A (using T as a treatment variable) and in appendix B (using T2 as a treatment variable)

23
impact in PEC and non-PEC schools. The DID estimator suggests that PEC has had a significant reduction in drop-out rates in PEC schools (row 3), no significant effect on failure rates (row 8) and a negative but insignificant effect on repetition rates (row 13).

Generally similar patterns emerge when we use a less strict definition of treatment. Receiving a PEC grant in any of the three school years (i.e., using T2 as the measure of treatment) lowers drop-out rates by 0.093 percentage points (row 3) and repetition rates by 0.034 percentage points (row 13).

Next, we move on to the impact estimates obtained using propensity score matching (PSM).<sup>12</sup> The most significant determinants of PEC participation include being a nonindigenous school, having more rooms and classes, and not receiving compensatory programs from Mexico’s National Commission for Educational Development CONAFE. At the locality level, communities with more developed infrastructure are generally more likely to have PEC schools.

The DID estimates of program based on PSM reveal considerably larger impacts relative to the regression-based DID impact estimates. Using PSM, we find that participation in PEC decreases drop-out rates by 0.24 percentage points (row 3), failure rates by 0.24 percentage points (row 8) and repetition rates by 0.31 percentage points (row 13). In addition, all these effects are statistically significant. Using the less strict definition of treatment (i.e. T2) yields impact estimate that are lower, but these continue to be significant. One immediate conclusion that can be derived from these findings is that method matters. The fact that the counterfactual constructed by the PSM method weights observations differently than the regression approach and uses non-PEC school that are very as similar to PEC schools as possible, makes a big difference on the estimated impact of the program. Our comparisons highlight the well known fact in the evaluation literature that the

> 12 The probit estimates of the determinants of participation in PEC are contained in Appendix C (table C.1 for T and table C.2 for T2). We do not report BA estimates of program with PSM as these are redundant.

24
choice of the comparison group is one of the most important aspects of a credible impact evaluation.

One might hypothesize that the impact of PEC would vary between types of schools and communities, so Table 6 presents estimates of impact allowing for heterogeneity across types of schools and localities. These estimates are obtained using T2 as the treatment variable, which allows for a larger number of schools in the treatment group. Magnitude and precision of estimates vary across these sub-samples, partly due to the different sample sizes and also perhaps due to PEC having heterogeneous treatment effects. PEC has the expected effect in most sub-samples: for each outcome, only two of the ten sub-samples associate PEC with increased dropout, repetition, or failure, and none of these positive associations has statistical significance. In most others, PEC is associated with decreased dropout, repetition, and failure, though with greater precision in some populations than others. PEC is found to have no statistically significant impact on the dropout, repetition, and failure rates in the 459 indigenous schools that implemented PEC plans. The largest impacts and most precisely estimated impacts are observed for non-indigenous schools and schools in urban localities. Impact does not have a monotone relationship with a community’s level of development—in very low marginality communities, where Mexico’s better-off citizens live, PEC has statistically insignificant impact on dropout but decreases failure and repetition by statistically significant 0.26 and 0.22 percentage points, respectively. But in higher marginality communities, PEC has in some cases larger and in some cases smaller estimated impact.

# **7. Conclusions**

25
Increasing students’ acquisition of cognitive skills can increase lifetime earnings, expand basic capacities, and potentially equalize a country’s distribution of income. Nonetheless, the many studies examining policy interventions aimed to improve schools find few that improve quality of schooling, measured by enrollment or test score outcomes. We measure the impact on dropout, repetition, and failure rates of Mexico’s five-year Quality Schools Program (PEC). PEC combines increased resources for schools with decentralization to allow school principals to make management decisions using the increased resources.

PEC has some similarity to public programs in many countries and U.S. states which give grants for local authorities to allocate (“school-based management”). Unlike some of these programs, PEC requires schools to design an improvement plan, involves parents in budget and planning decisions but does not let parents make personnel decisions, and PEC provides a short-term grant to schools and trains school principals while other programs do not. Since every PEC school receives these interventions as a package, we cannot distinguish their impact, but we can examine the impact of PEC as a whole.

We construct a panel of 74,700 schools and use two common non-experimental methods to create a control group and estimate program impact: regression analysis and propensity score matching. We compare the estimated impact of PEC on dropout, repetition, and failure rates using these two methods. The preferred estimator, difference-in-differences with matching, reveals that participation in PEC significantly decreases dropout rates by 0.24 percentage points, failure rates by 0.24 percentage points and repetition rates by 0.31 percentage points. The estimated impacts are slightly lower but remain statistically significant when participation in PEC is measured as receiving PEC grants for any one of the three school years covered by our study. PEC is found to have no significant impact on the dropout, repetition, and failure rates in indigenous schools.

This impact has moderate magnitude—it represents a decrease of 6 to 8 percent relative to the baseline mean levels of dropout, repetition, and failure. Given Mexico’s 26
primary school enrollment of 14.8 million students in 2000 (SEP 2000), these estimates suggest that if PEC had operated in all schools in the year 2000 and had the estimated mean impacts, 35,500 Mexican primary school students would not have repeated a grade or dropped out of school, and 46,000 students would not have failed a grade. Students who drop out, fail, or repeat generally represent Mexico’s poorer and more disadvantaged students, so for these outcomes PEC enhances equity across students. These represent a fairly large group in absolute magnitude, though a small group relative to all Mexicans.

Our comparisons highlight the well known fact in the evaluation literature that the choice of the comparison group is one of the most important components of a credible impact evaluation. The fact that the counterfactual constructed by the propensity score matching method weights observations differently than the regression approach and uses non-PEC school that are very as similar to PEC schools as possible, makes a large difference on the estimated impact of the program.

The results suggest that PEC’s combination of increased school resources and local school management can produce small but statistically significant improvements in learning. The smaller sample for indigenous schools, which represent some of Mexico’s poorest communities with the worst education outcomes, may contribute to the larger standard errors in estimates of PEC impact these communities. But the smaller impact estimates for indigenous schools may reflect that PEC has less impact in them. This finding would cohere with others (Galiani et al., 2004) showing that decentralization of public management can improve outcomes in wealthy areas but has smaller or even negative impact in the most disadvantaged areas.

Dropout, repetition, and failure rates represent good but not complete outcomes - measuring impact on test scores, completion rates, parental involvement in schools, and school autonomy would give a more complete picture of PEC’s impact. Collecting data on 27
these indicators that is comparable between PEC and non-PEC schools and over several years would facilitate such analysis.

We conclude by emphasizing an important caveat. Our estimates of program depend critically on the validity of our maintained assumption that selection into the program is based on observable and time invariant unobservable variables. Including preintervention trends in outcomes in addition to an array of covariates gives some confidence in the validity of our estimates. Our preferred estimator, the differences-in-differences matching estimator, while an improvement over simple before-after and cross-section comparisons, still suffers the weakness that it ignores unobserved factors which change over time between PEC and non-PEC schools. The extent to which these factors result in biased estimates of program impact is something that can be most easily by applying experimental methods.

28