metadata: authors: 0: author: De Hoyos Navarro,Rafael E.
1: author: Garcia Moreno,Vicente A.
2: author: Patrinos,Harry Anthony
count: Mexico
docna: 0: docna: The impact of an accountability intervention
            with diagnostic feedback : evidence from Mexico
docty: Policy Research Working Paper
subtopic: Educational Sciences,Educational Institutions & Facilities,Educational Populations,Education for Development (superceded),Effective Schools and Teachers,Education For All
teratopic: Education
abstracts: cdata!: In 2009, the Mexican state of Colima
            implemented a low-stakes accountability intervention with
            diagnostic feedback among 108 public primary schools with
            the lowest test scores in the national student assessment. A
            difference-in-difference and a regression discontinuity
            design are used to identify the effects of the intervention
            on learning outcomes. The two alternative strategies
            consistently show that the intervention increased test
            scores by 0.12 standard deviations only a few months after
            the program was launched. When students, teachers, and
            parents in a school know that their scores are low, and this
            triggers a process of self-evaluation and analysis, the
            process itself may lead to an improvement in learning
            outcomes. Information on quality, without punitive measures
            but within a supportive and collaborative environment,
            appears to be sufficient to improve learning outcomes.
geo_reg_key: 517191,119228
repnme: repnme: The impact of an accountability
            intervention with diagnostic feedback : evidence from Mexico
display_title: The impact of an accountability
            intervention with diagnostic feedback : evidence from Mexico
listing_relative_url: /research/2015/08/24903688/impact-accountability-intervention-diagnostic-feedback-evidence-mexico
topicv3: School Directors m1326118 2144,Teachers Management m1326119 1523,Quality of Education m1326344 491,Education m1327048 470,School-Based Management m1327730 2143
publishtoextweb_dt: 2015-08-13T00:00:00Z
historic_topic: Education
no_of_pages: 36
author_keys: 0: author_key: 900926
1: author_key: 713850
2: author_key: 95903
model_extractions: WPS7393 Policy Research Working Paper 7393 The Impact of an Accountability Intervention with Diagnostic Feedback

# Evidence from Mexico

_Rafael de Hoyos Vicente A. Garcia-Moreno Harry Anthony Patrinos_ Education Global Practice Group August 2015
Policy Research Working Paper 7393

# **Abstract**

In 2009, the Mexican state of Colima implemented a low-stakes accountability intervention with diagnostic feedback among 108 public primary schools with the lowest test scores in the national student assessment. A difference-in-difference and a regression discontinuity design are used to identify the effects of the intervention on learning outcomes. The two alternative strategies consistently show that the intervention increased test scores by 0.12 standard deviations only a few months after the program was launched. When students, teachers, and parents in a school know that their scores are low, and this triggers a process of self-evaluation and analysis, the process itself may lead to an improvement in learning outcomes. Information on quality, without punitive measures but within a supportive and collaborative environment, appears to be sufficient to improve learning outcomes.

This paper is a product of the Education Global Practice Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at rdehoyos@ worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# The Impact of an Accountability Intervention with Diagnostic Feedback: Evidence from Mexico<sup>_∗_</sup>

Rafael de Hoyos<sup>_†_</sup> Vicente A. Garcia-Moreno<sup>_‡_</sup> Harry Anthony Patrinos<sup>_§_</sup>

## **1 Introduction**

The increasing availability of standardized tests in developing countries opens the door to a wide range of policy interventions to improve student learning outcomes. Under certain conditions, the implementation of standardized tests in itself can be conductive to an increase in learning outcomes. When test scores are made public, they can eliminate heterogeneous perceptions among agents in the system (authorities, school directors, teachers, parents and students) regarding the quality of education services. Given sufficient levels of initial asymmetries on perceptions regarding the quality of education, the additional information brought about by test scores can be enough to generate a new equilibrium yielding a more transparent and accountable system providing higher quality education services.

The second quality-enhancing change that can be triggered by the availability of test scores is the development of school-specific improvement plans. If the standardized tests are universal, then every school in the system could have access to a detailed diagnosis of the main challenges in the subject areas and grades assessed by the tests. If these detailed diagnoses and school improvement plans are followed by what can be loosely defined as _pedagogical interventions_ , or actions to address the problems identified, then the final outcome can be a system delivering higher quality services.

> _<u>∗</u>_ The authors gratefully acknowledge funding from the World Bank’s _Research Support Budget_ and

> the comments of participants at the World Bank’s economics of education seminar.

> _†_ The World Bank. E-mail: rdehoyos@worlbank.org.

> _‡_ Teachers College, Columbia University. E-mail: vag2120@tc.columbia.edu

> _§_ The World Bank. Email: hpatrinos@worldbank.org
There is evidence, mostly from high-income countries, showing that learning outcomes can improve as a result of an intervention increasing transparency and accountability through the use of test scores. For instance, in the Netherlands, both average grades and the number of diplomas awarded increased after schools received a negative report card. For schools that received the lowest ranking, the one-year effects on final exam grades amounted to 10 to 30 percent of a standard deviation (Koning and van der Wiel, 2013). Evidence shows that schools in the United States respond to accountability pressures with improved test scores (Carnoy and Loeb, 2003; Hanushek and Raymond, 2005). Students in high-accountability states averaged significantly greater gains on 8th grade math tests than did students in states with little or no state measures to formally track student performance. One of the most prominent examples of accountability interventions is the state-level ranking system of No Child Left Behind (NCLB) in the United States. Rockoff and Turner (2010) found that the introduction of student tests and other measures to assign each school a grade was enough to increase student achievement in New York City. Rouse et al. (2013) show that schools facing accountability pressures in Florida changed instructional practices and this, in turn, partly accounted for increases in test scores. Using administrative data for the entire country, Ahn and Vigdor (2014) find evidence of a short term positive impact on school performance, and among low performing students in the medium term.

In most of the documented cases of education _accountability interventions_ in high-income countries, school actors respond to this type of intervention when they are followed by rewards and/or sanctions. However, in low- and middle-income developing countries with a combination of relatively weak institutions, high poverty and inequality levels, and the presence of powerful teacher unions, such high stakes accountability interventions might not be feasible nor desirable. Instead, a shared responsibility approach characterized by a supportive and collaborative environment rather than a punitive one might prove to be a more effective accountability intervention. Although low-stakes accountability interventions like the one describe here could work through implicit mechanisms, such as stigmatization or reputational damage, they could also have an impact through coordinated collaboration among school actors and pedagogical tools to sustain a shared responsibility intervention. Identifying the schools as low performing, meeting with them, developing a detailed diagnosis to identify their main challenges, and offering them advice to design a school-specific improvement plan, could be enough to improve service delivery.

The evidence on the effects of low-stakes _accountability interventions_ within a supportive and collaborative environment in developing countries is mixed. The experimental evidence for India presented in Muralidharan and Sundararaman (2010) shows that a program that provided low-stakes diagnostic tests and feedback to teachers had no effect 2
on student learning outcomes. In a randomized study in Punjab, Pakistan, Andrabi et al. (2014) show that providing test scores to households and schools leads to increases in subsequent test scores by 0.11 standard deviations after one year of the intervention. Test score gains in public schools were in response to a low-stakes threat since there were no formal consequences attached to results. Public schools in Punjab face little competitive or regulatory pressure to perform, yet the information has a significant impact because there are non-monetary mechanisms such as social/community pressures on public school teachers that induce performance improvements if a school is revealed to have low test scores. In Latin America, the results presented in Mizala and Urquiola (2013) show that distributing information regarding schools’ value added in Chile had no effects on enrollment, tuition levels or socioeconomic composition of students suggesting a limited effect of a low-stakes accountability intervention.

This paper evaluates the impact on learning outcomes of a short-lived program suitable to measure the effect of a low-stakes accountability intervention within a supportive and collaborative environment and, albeit, in a more limited way, the _pedagogical interventions_ effect. The program, PAE, short for _Programa de Atenci ́on Espec ́ıfica para la Mejora del Logro Educativo_ was implemented in the Mexican state of Colima between January 2010 and late 2011, with the objective of increasing learning outcomes among the worst performing public primary schools in the state. Although originally designed as a comprehensive schooling intervention with various components, PAE was cut short administratively and in the end only a subset of the components were implemented. Schools were informed of their test scores and that they were to be part of the program; once the program was launched and the list of PAE schools was publicly disseminated, participating schools were assigned a technical adviser who would visit the school and help diagnose the test score results and design a school improvement plan.

The present study follows two alternative strategies to identify the effects on PAE on learning outcomes. The first one follows a difference-in-difference approach comparing the evolution in test scores between PAE and non-PAE schools over time. The second strategy exploits PAE’s rigid eligibility rule (an exogenously determined cut-off point of the national standardized test) dividing schools into treatment and control groups and compares them through a regression discontinuity design estimating the accountability effect of PAE. Both strategies consistently show that the intervention increased test scores by 0.12 standard deviations only a few months after program launch. Although this effect remains one and, to a lesser extent, two years after program implementation, our results show no additional impact attributable to the implementation of the _pedagogical interventions_ .

The intuition behind these results is that public recognition of low performing schools, together with a detailed diagnosis of the school’s main challenges and an invitation to net3
work with other school directors, teachers and advisers to develop improvement strategies is enough to improve the quality of education services. In fact, no additional inputs to the schools were actually funded or provided within the first few months of the program. In this way, the current study examines how standardized tests were used to identify poor performing schools and how merely diagnosing school problems and developing a school improvement plan can lead to higher student performance. One way that the accountability effect results in quality improvement is that it creates a fact-based dialogue among educational authorities, school principals and teachers. In this way, identification of the schools’ main problems within a supportive and collaborative environment is not only the first step toward improvement, it is arguably, in itself, sufficient for improvement.

The paper is organized as follows: Section 2 provides background information on the Mexican state of Colima, describes the PAE, and charts some trends in learning outcomes. Section 3 details the methodology and identification strategies while Section 4 discusses the main results. Finally, a concluding Section 5 enumerates some policy recommendations.

# **2 Background, PAE Description and Recent Trends**

Colima is a small state in the center-west region of Mexico, with 650,000 inhabitants, 34 percent of whom live in households with incomes below the official 2012 poverty line.<sup>1</sup> Since the decentralization of the education system that began in 1992, Colima has built an efficient school system adjusting the national educational programs to the state’s specific characteristics and needs. Throughout the 1990s, Colima undertook innovative education policies such as the implementation and dissemination of one of the country’s first standardized tests. By 2003, Colima outperformed all Mexican states in the OECD’s PISA test and actually approached the OECD average. For example, Colima’s math scores were on par with those for Greece and Serbia, and higher than those from Thailand, Brazil, Uruguay and Turkey.

At the end of school year 2005-2006, for the first time, the Federal Ministry of Education (SEP) applied a universal standardized test (ENLACE). ENLACE, a low stakes test, gathered information on student performance in math, Spanish and a rotating subject for third, fourth, fifth and sixth graders in private and public primary schools in Mexico. By design, ENLACE has a national mean score of 500 and a standard deviation of 100 for every subject area and grade. In early October 2009, the results from ENLACE 2008-2009 were published, and Colima performed below the national average. All the schools in Colima had access to this information, but there is no evidence of rankings

> 1The poverty headcount ratio in Mexico was 45.4 percent in 2012.

4
or public dissemination of the results at this stage. A few weeks after the release of the disappointing ENLACE results, Colima’s Ministry of Education (CMoE) began the design of PAE.

# **2.1 PAE Description**

PAE was a program designed to improve learning outcomes among the lowest performing public primary schools in Colima, which also provide education services to students from the most marginalized schools ((Mizala et al., 2007)). The program’s operational rules excluded multi-grade schools with one or two teachers and community schools managed by the _Consejo Nacional de Fomento Educativo_ (CONAFE).<sup>2</sup> Of the 477 primary schools in Colima in academic year 2008-2009, 40 were private, 39 were managed by CONAFE, 78 were one- and two-teacher schools, and 10, for a variety of reasons, did not have an ENLACE score. The group of PAE-eligible schools consisted of 310 public primary schools (see Figure 1) with a total student population of 62,366 (95.2% of the total number of students in public primary schools in Colima during the 2008-2009 school year). Between October and November 2009, the CMoE used the 2008-2009 ENLACE score data to construct its ranking of schools. School scores were a simple average of the three subject areas tested: math, Spanish and science across grades 3, 4, 5 and 6. Schools in the 35th percentile or less of the distribution of the school average test scores were automatically designated as PAE schools (see Figure AI in Annex). As shown in Figure 1, PAE included 108 of the 310 schools that belong to the potentially eligible population.<sup>3</sup> PAE schools were distributed across all ten municipalities of Colima and encompassed 1,091 teachers and 10,550 students in 2009.

Figure 2 illustrates the timeline followed by the design and implementation of Colima’s PAE. Between November and December 2009, the CMoE assigned the schools that were going to participate in PAE following the criteria described above. In January 2010, the selected schools were officially notified. In February 2010, at a teachers’ congress in Colima, the Governor launched the program and publicly disseminated the list of PAE schools. Although the assigned schools were presented as those with the lowest learning outcomes in the state, the Governor emphasized the co-responsibility behind low performance of schools and state education authorities and the importance of working together to make improvements. Between the public announcement of the program and the first follow up ENLACE test in May 2010, which the CMoE called Phase 1 or the “awareness” period, PAE schools were assigned a _technical adviser_ , part of the CMoE, who would visit the school three times a month to work with school directors and teachers on

> 2For details on CONAFE see www.conafe.gob.mx.

> 3Two schools were dropped from the sample due to a mistake in their original classification as nonmulti-grade schools which later on was changed to multi-grade.

5
the diagnosis of the ENLACE test and the design of improvement strategies. In addition, the PAE technical adviser coached teachers on analyzing the ENLACE information to have a clearer understanding of how schools were assigned to PAE and the causes of poor performance within their schools.

Between January and March 2010, authorities at Colima’s Ministry of Education, together with school directors and selected teachers, developed a simple methodology to construct a detailed diagnosis identifying the academic weaknesses of their students based on ENLACE results.<sup>4</sup> The diagnoses were tailored to each school: the ENLACE test questions which more students answered wrong were collected by subject area, grade and classroom. Using personal identification numbers and a password, all teachers in Mexico had online access to a rich data set organizing the proportion of students in their class who answered an ENLACE question incorrectly. The website also indicated the area of knowledge and the relevant curriculum area for each ENLACE question, thereby providing teachers with concrete pedagogical direction to guide their efforts (see Figure 3 showing an example of the type of information provided by ENLACE).

Between February and May 2009, PAE’s technical advisers, school directors and teachers worked on a school improvement plan to address the problems identified during the diagnosis. The school improvement plan had to include clear medium- and long-term goals in terms of learning outcomes and a plausible strategy, involving teachers and parents, to reach them. The CMoE’s technical advisers would visit PAE schools three times a month to follow up on the implementation of the school improvement plans. However, the “awareness” period of the program was too short to change any of the fundamental inputs of the learning production function and it is, hence, capturing the accountability and diagnosis effect of the program (see Figure 2).

The second phase, or the implementation stage, of PAE started in September 2010. It consisted of pedagogical interventions and the monitoring of progress. With diagnoses and school improvement plans in hand, state authorities, school directors and teachers collaboratively implemented the school-specific improvement strategies which, broadly speaking, included one or more of the following four interventions:

1. Strengthening school-based management. This intervention draws on the experiences gained from the on-going national school based management program: PEC ( _Programa Escuelas de Calidad_ ) and AGE ( _Apoyo a la Gesti ́on Escolar_ ). It should be noted that all schools in Colima at one time or another participated in PEC or PEC-like program.

2. Redefining the role of school supervisors and providing them with training. In 2010, school supervisors in Mexico were not appointed through a competitive process and

- 4The methodology relied on public information generated by the Federal Ministry of Education (SEP).

6
did not have to undertake any training before taking on their duty. Hence, there was a high degree of variation in the quality of school supervision.

3. Specifying the role of school directors and providing them with training. Similar to school supervisors, many of the school directors lack the necessary skills on how to manage a school. Directors had difficulty identifying the strengths and weaknesses of their school, and in turn were limited in their capacity to design and monitor school improvement plans. For example, school directors seldom set measurable and reachable goals in crucial indicators to monitor progress.

4. Reinforcing teachers’ knowledge in the identified academic areas posing challenges. The program provided teachers with training and special courses to strengthen their knowledge in subject areas identified as challenging during the diagnosis step of the program.

Due to reasons unrelated to the program’s performance, PAE was canceled in December 2011, one year after the pedagogical interventions were in place. A difference-in-difference and a regression discontinuity approach is used to answer the following research questions:

1. Does PAE increase learning outcomes among participating schools?

2. If there is a positive effect of PAE, then is this explained by the accountability plus diagnosis intervention or by the pedagogical intervention or both?

3. Did PAE have a differentiated effect across boys and girls or among students with relatively disadvantaged initial conditions?

# **2.2 Dataset and Recent Trends**

This study uses and merges student learning outcomes as measured by ENLACE with administrative school census data collected by federal and state education authorities (known as the _Formato 911_ ). Since 1998, this school census is collected at the beginning and end of each school year, and lists, among other entries, the number of teachers, students, classrooms, computers, the average years of education of teachers, and the geographic location of each school. With a unique school identifier ( _Clave de Centro de Trabajo, CCT_ ), it is possible to merge this school census data with the results from ENLACE into a single data base. In addition to learning outcomes, ENLACE includes socioeconomic information for each school based on their geographical location.<sup>5</sup>

> 5The National Population Council (Consejo Nacional de Poblaci ́on, CONAPO) ranks all localities (an administrative and / or geographic entity often more disaggregated than a municipality) in Mexico according to a marginality index, a weighted average of literacy, access to basic public utilities, household infrastructure and average wages. Rankings range from very high marginalization, high marginalization,

7
Figure 4 shows mean math scores in PAE and non-PAE schools from 2006 to 2013. In general, schools in Colima have improved by 42 points of ENLACE, or 0.42 standard deviations, throughout this period. As expected, the 108 PAE schools had lower learning outcomes relative to the non-PAE schools. In 2009, the baseline year, the difference between PAE and non-PAE schools was 56 points; the gap was reduced to 42 point the year after, perhaps partly explained by mean reversion effects, but also explained by the effects of PAE (see (Chay et al., 2005)).

# **3 Methodology**

We follow two alternative methodologies to identify the effects of PAE on student-level learning outcomes: a differences-in difference (DD) approach exploiting differentiated performance on learning outcomes between PAE and non-PAE schools and a regression discontinuity (RD) approach exploiting the exogenously defined threshold or cutoff point dividing PAE and non-PAE schools. The DD identifies the effects of the program assuming that performance on learning outcomes, in PAE and non-PAE schools, would have been homogeneous in the absence of the program, which could be a restrictive assumption. RD overcomes this limitation by exploiting differences in performance between PAE and non-PAE schools in the neighborhood of the exogenously determined cutoff point defining PAE schools. However, the limitation of RD is that the number of clusters (in our case schools) around the cutoff needed to identify a given effect under RD is relatively large, hence reducing the precision of the estimates. Therefore, getting similar effects under the two alternative strategies provides more robust evidence of the true impact of PAE on learning outcomes.

Formally, let us define _Yi,s,t_ as the test score of the _ith_ student in school _s_ in year _t_ and _PAEs_ as a dummy variable taking the value of one if the school is part of the program, zero otherwise. The DD is estimated via the following regression using OLS: where _A_ are year fixed effects, _Xs,t_ are a series of school-level controls and _εi,s,t_ is a random component. The parameters of interest capturing the impact of PAE are the ones measuring the learning outcome effects of the interaction between _PAE_ and the year fixed effects, _λt_ . If these parameters are statistically significant they would indicate that medium marginalization, low marginalization, and very low marginalization. For methodological details regarding Mexico’s marginality index, see www.conapo.gob.mx 8
PAE schools performed differently from non-PAE schools, controlling for everything else, suggesting that the program had an effect on learning outcomes.

There could be several reasons why an identification strategy such as the one described by equation (1) would yield biased estimators. First, the two groups, PAE and nonPAE schools are not equal ex-ante. Therefore, factors unrelated to the program can impact both groups differently causing a different post-PAE performance among treated and untreated schools and hence biasing the DD results. Second, the selection of schools on the basis of a one-year school performance ranking may misclassify schools due to a one-time performance aberration (one time shocks or mean reverting noise). As discussed by Chay et al. (2005), this would produce biased estimators of the program’s impact since PAE (low-performing) schools would tend to automatically revert to the overall mean.

According to Chay et al. (2005), a regression discontinuity design can defuse these two identification problems. The logic behind the RD approach is simple: the objective is to identify a group of schools that are part of PAE and similar enough to a group of schools that are not part of the program. A good place to identify such comparison groups is around the cutoff point distinguishing PAE from non-PAE schools as the threshold mimics a randomized selection to receive or not to receive treatment (Imbens and Lemieux, 2007; Imbens and Wooldridge, 2008). Formally, let us define _Yi,s,t_ as a function of _PAE_ , the average results of school _s_ at the baseline _Ys,_ 2009, the interaction between the former and the latter, a series of school-level controls _Xs,t_ and random component _ε_<sup>_′_</sup> _i,s,t_<sup>:</sup> Notice that the dummy variable identifying schools belonging to PAE and their eligibility variable, ENLACE average results for 2009, are constant over time. By assumption _ε_<sup>_′_</sup> _i,s,t_ should be independently and identically distributed (iid) with a mean of zero and known variance. Equation (2) can be modified to include higher order terms of the _forcing variable_ , _Ys,_ 2009, to control for non-linearities in the relationship between the eligibility criteria and subsequent learning outcomes.<sup>6</sup> For a group of schools sufficiently close to the PAEeligibility cutoff, such that samples are balanced both in observables and unobservables, the effects of PAE will be captured by _β_<sup>ˆ</sup> 1 _,t_ in Equation (2). An important limitation to the regression discontinuity design, however, is that the results are valid only for observations around the cutoff point; the estimated impact is limited to a local average treatment

> 6Higher order polynomials are typically used when estimating RD using all the available information as oppose to restricting it to those observations within the optimal bandwidth (see (Imbens and Lemieux, 2007)).

9
effect which cannot be generalized to cover the entire population, thereby undermining the external validity of the estimation. A second limitation more relevant for the current study is that RD relies on having a large number of observations around the cutoff. As shown by Dragoset and Deke (2012) and Schochet (2005) for a constant statistical power and minimum detectable effect, the number clusters (schools) needed under RD is substantially larger than the one needed under a randomized control trial, hence, limiting the statistical power in the evaluation of PAE.

Under both approaches, DD and RD, the unit of intervention is the school but the unit of analysis is the student (that is, schools, not students, are assigned to PAE). Therefore, the unobservables are composed of two terms _εi,s,t_ = _ηs_ + _νi,s,t_ . In other words, the unobservables are composed of a school-specific component ( _ηs_ ) and an individual-, school- and time-specific term ( _νi,s,t_ ). This structure of the error term implies clustering of students within schools allowing for intra-school correlation across students. Since PAE started in January of 2010 and the first follow up ENLACE test was a few months later, in May 2010, the DD estimator _λ_<sup>ˆ</sup> 2010 and RD estimator _β_<sup>ˆ</sup> 1 _,_ 2010 capture the accountability effect of the program while those for the following years capture the effects of the pedagogical interventions.

# **3.1 Determining the bandwidth of comparable schools**

The optimal number of schools around the cutoff by which to evaluate the impact of the PAE program is determined by a trade-off between precision and internal validity. That is, a narrow bandwidth would select schools very close to the cutoff, hence more similar in observables and unobservables, but the statistical power might be compromised given the small number of observations. On the other hand, a wider bandwidth would increase the number of observations in the treatment and control groups but might not yield balanced samples in observables (and unobservables). We follow the method developed by Imbens and Kalyanaraman (2012) to determine the optimal bandwidth which yields 20.2 ENLACE points below and above the cutoff or 0.202 standard deviations around the threshold dividing PAE from non-PAE schools. This optimal bandwidth will be complemented with two alternative but rather arbitrary bandwidth: half of the optimal bandwidth ( _±_ 10.1 points of ENLACE) and double the optimal bandwidth ( _±_ 40.4 points of ENLACE).<sup>7</sup> Table 1 shows the number of schools above (non-PAE) and below (PAE) the cutoff as well as the number of students using each of the three different bandwidths.

Figure 5 shows the intra-cluster correlation for those schools around the cut-off: in other words, the proportion of total variation in student-level math learning outcomes that is

> 7The optimal bandwidth was computed using the regression discontinuity Stata program RD developed in Nichols (2014).

10
accounted for by differences between schools. This coefficient is practically zero until 10 ENLACE points around the threshold, and less than 0.02 at 20 ENLACE points (the optimal bandwidth) above and below the cutoff. The relatively low value of the intra-cluster correlation among schools within the optimal bandwidth presented in Figure 5 suggests that a valid comparison between schools with 0.2 standard deviations of the cutoff can be conducted. In a ranking where first place indicates the lowest scoring school, schools ranging from 41 to 108 are defined as the PAE treatment group while the non-PAE control group would consist of schools 109 to 171. A second requirement for the RD approach to be valid is that the density of the _forcing variable_ must be continuous around the cutoff and this is what is shown by Figure AI in the Annex.

Granted sufficient observations around the cutoff, the RD approach may mimic a randomized experiment if the treatment and control groups are equal in expectation on all observed and unobserved dimensions. Table 2 shows school inputs in the school year 2008-2009 (the baseline) for schools within the optimal bandwidth. School inputs are statistically equal across treatment and control. Differences in school size, number of teachers, the proportion of teacher that are part of a monetary incentives program ( _Carrera Magisterial_ ), the proportion of teachers with a university degree or higher, as well as the dropout and failure rates between PAE and non-PAE schools within the optimal bandwidth are statistically insignificant. One of the six variables compared during the baseline is statistically different between PAE and non-PAE schools, indicating that PAE schools tend to be poorer that non-PAE, even when comparing only those located _±_ 20.2 points of ENLACE around the cutoff.

# **4 Results**

This section shows the results of the DD and RD approaches described above focusing on the effects of PAE on math test scores. The effects on Spanish are listed in the Annex as a comparison. The results include estimations of DD and RD with and without school-level controls using all available data and restricting the estimation for those schools within the optimal bandwidth. The DD are also estimated using only schools around the cutoff to make them more comparable to the RD estimates and to make the results robust to differences in initial conditions. In all estimations standard errors are clustered at the school level.

11
# **4.1 DD Approach**

Table 3 presents the DD effects of the PAE program on math learning outcomes using different specifications. The first of these specifications (column 1 in Table 3) estimates the DD effects without controls and using all available data of 311 PAE-eligible public primary schools in Colima. The results show an effect of almost 0.13 standard deviations ( _σ_ hereafter) in 2010, a few months after PAE was launched. According to this simple specification, the DD estimator increased over time although the estimated effects in 2011 and 2012 are not statistically different from that in 2010, as suggested by the learning outcome trends shown in Figure 4. Notice that specification (1) with standard errors clustered at the school level, yields very similar effects to a specification with school fixed effects; that is, a dummy variable for each of the 311 eligible public primary schools (clusters) in Colima. Specification (2) includes the following school-level controls: student - teacher ratio, proportion of teachers that are enrolled in the monetary incentives program _Carrera Magisterial_ , proportion of teachers with a university degree or a post-graduate diploma, and the level of marginalization of the locality where the school is located. Including these controls, which are highly significant, reduces the estimated effects of PAE to close to 0.10 _σ_ in 2010 but still significant at the 99 percent confidence level.

Specifications (3), (4) and (5) are the same as specification (2) but restricting the observations to schools within the optimal bandwidth (OB), half of the OB, and twice the OB. Not surprisingly, under the OB restricted sample, the controls reduce their significance, particularly the proxies for socio-economic status (marginalization levels). However, the effects of PAE remain statistically significant with a point estimator close to 0.11 _σ_ . Notice that under specification (3), the DD estimators across years 2010, 2011 and 2012 are very similar suggesting that, between 2009 and 2010, PAE schools managed to improve 0.11 _σ_ faster vis-a-vis the changes in comparable non-PAE schools as a result of the intervention, capturing the _accountability_ effects of the program and its subsequent diagnosis activities during the “awareness” period. Restricting the schools included in the analysis to half of the OB does not change the DD estimator; however, with only 67 clusters, the estimates lose precision. DD estimations using schools within double the size of the OB (222 clusters) yields very similar results as those with the OB. Albeit marginally smaller, the effects of PAE on math scores are corroborated by the program’s effects on Spanish (see Table AI in Annex). The effects on Spanish scores in 2010 range from close to 0.12 _σ_ when estimating the DD without controls and using all available information, to 0.09 _σ_ when the controls are included and 0.08 _σ_ when restricting the estimation to schools within the OB, always significant at the 95 percent confidence level.

Bertrand et al. (2004) show that, by ignoring the fact that DD focuses on serially correlated outcomes, its estimation can severely underestimate standard errors. They show that a simple correction consisting of collapsing the time series information into a “pre”- and 12
“post”-intervention period would be enough to account for this time series inconsistency in standard errors. Tables AII and AIII in the Annex shows that when the DD is performed using a sample with data collapsed into two periods, before and after PAE, the results are practically unchanged. The effects on math (Spanish) test scores range from 0.18 _σ_ (0.14 _σ_ ) with the full sample and no controls to 0.14 _σ_ (0.09 _σ_ ) when the controls are added and 0.13 _σ_ (0.08 _σ_ ) when the sample is restricted to schools within the OB, in all cases significant at the 95 percent confidence level.

As mentioned in Section 3 one of the most important limitations of the DD approach as a strategy to evaluate the impact of PAE is that program effects can be confounded with mean-reversion noise caused by a treatment selection based on ranking of schools at the baseline. Although the RD is a robust strategy to address this concern, we also perform a simulation showing that mean-reversion is not driving the results presented so far. If mean-reversion is at work, then reproducing PAE’s eligibility criteria but using the ENLACE results of 2008 (as opposed to 2009) to rank and select schools should result in PAE schools advancing faster than non-PAE schools between 2008 and 2009. Figure AII in the Annex shows the average evolution of learning outcomes between PAE and non-PAE schools using the alternative selection criteria based on the school ranking of 2008 (top figure) and compares it with the evolution under the actual selection of PAE and non-PAE schools. The simulation and actual trends on average test scores in Figure AII are based on schools within the OB. Using the ranking of 2008 to assign schools into PAE and non-PAE produces no differentiated evolution in test scores after the assignment, suggesting that the post-2009 PAE versus non-PAE differences in trends observed after 2009 can be attributable to the program with little or no mean-reversion effects.

# **4.2 RD Approach**

The first strategy followed within the RD approach is a graphical representation of the discontinuity using local linear or kernel regressions on both sides of the cutoff. Figure 6 illustrates the relationship between average math performance at the school level in 2010 (vertical axis) and the _forcing variable_ , the simple average ENLACE results at the school level in 2009 relative to the test score (horizontal axis). The PAE schools are to the left of the cutoff, which was used for their eligibility into the program, and the non-PAE schools are to the right. A mild discontinuity appears at the cutoff (469 points of ENLACE): there is a general pattern showing that schools slightly below the cutoff (the PAE schools) display greater test scores in 2010 than schools slightly above the cutoff (the non-PAE schools), although their scores in 2009 were very similar. Zooming into the discontinuity reveals that the difference in the regression at the cutoff is around 10 ENLACE points, very close to DD estimates. This graphic illustration suggests a positive effect on learning 13
outcomes in 2010 brought about by the program and is consistent with what was shown by the DD approach.

Analyzing the impact of the program after one year, when the schools may have implemented some of the pedagogical interventions, provides additional information on the impact of the PAE. The graphical representation of the RD in 2011, shows no discontinuity at the cutoff. Schools on the left of the threshold have an average achievement in 2011 similar to those schools on the right of the cutoff (see Figure 7). Similar results showing no-discontinuity are found in 2012, two years after the implementation of the program.

The parametric estimation of RD, equation 2, includes the same specifications as the DD approach. In other words, the results presented in Table 4 include a specification using all available information (more than 38,000 students in 311 schools at baseline) and including school-level controls while specifications 2, 3 and 4 restrict the observations to those schools within different bandwidths. All standard errors are clustered at the school level. According to the RD results of a specification using all available schools, PAE had an effect of 0.16 _σ_ on math learning outcomes in 2010 (with _ρ_ = 0 _._ 075), a few months after the implementation of PAE. These results are similar in magnitude to the ones obtained by the DD approach. However, the effect tends to disappear as we narrow the bandwidth of schools included in the regression. At double the OB (at _±_ 0.4 _σ_ around the cutoff) with 220 schools included, the effect of the program is 0.12 _σ_ significant at the 10 percent level; at the OB ( _±_ 0.2 _σ_ around the cutoff) with 127 schools included the effect is no longer significant; and at half of the OB ( _±_ 0.1 _σ_ around the cutoff) with only 67 schools included, the effect basically disappears.

The parametric estimation of RD for years 2011 and 2012 are shown in Tables AIV and AV in the Annex. Consistent with the graphic representation of the discontinuity, the parametric results for 2011 and 2012 show no effects of PAE on learning outcomes. These results combined with the significant effects for 2010 suggest that the impact of PAE on learning outcomes are driven by the _accountability_ and diagnosis effect with no apparent impact from the _pedagogical interventions_ . It seems that the diagnosis and design of a school improvement plan based on the results of ENLACE was enough to improve test scores among PAE schools; however, no further improvements were possible either because the _pedagogical interventions_ were not able to address the constraints faced by schools or because the interventions were not well implemented, or they simply needed more time to bear fruit.

The point estimator and standard error (SE) of PAE’s effect is significantly different between DD and RD, especially when we restrict the sample to the OB. While the DD and RD point estimates of 2010 are not substantially different when all the schools in the sample or those within double the OB are considered, the SE are significantly higher 14
under RD. For instance, taking the full sample and including controls, the RD shows an impact of 0.16 _σ_ with a standard error of 8.9, this last one being two a half times larger than the SE under DD which shows an effect of 0.10 _σ_ and SE of 3.5. As we restrict further the bandwidth under RD, the SE increases and the point estimate reduces. The increase in SE or lost precision of the RD estimator can be explained by poor statistical power attributable to the relatively few schools around the cutoff. Notice that although there is a large number of students in our sample, the number of clusters or schools is relatively small, especially when we restrict the sample to schools located within the OB (127 schools). Schochet (2005) estimated that, for the same level of statistical precision, the number of clusters needed under RD are three to four times larger than the sample size required under a randomized controlled trial (RCT). According to Schochet (2005) “[t]he reduction in precision in the RD design arises due to the substantial correlation, by construction, between the treatment status and score variables that are included in the regression models; this correlation is not present under the random allocation design.” Additional evidence in Dragoset and Deke (2012) shows that, under certain conditions, sample requirements are 9 to 17 times larger under RD that for RCTs. In our particular case, the number of clusters required under an RCT with an error of 0.05, power of 0.8, minimum detectable effect of 0.13 _σ_ , an intra-class correlation of 0.05, _R_<sup>2</sup> of 0.05 and an average number of students per school of 250 is equal to 200 clusters, or 100 treatment and control schools. Taking the results of Schochet (2005), to have the same statistical precision under an RD we would need at least three times more clusters, or 600 schools within the OB, a figure considerably higher the 129 schools within the OB. Therefore, the lack of precision under RD when we restrict the sample to schools within the OB is explained by a lack of statistical power due to the low number of schools around the PAE’s information dissemination strategy could have created incentives within schools that can explain a positive impact. First, the pressure put on PAE teachers by being declared low-performing schools may have created incentives to practice some type of “strategic behavior” by school directors or teachers such as cheating or teaching to the test (see (Figlio and Getzler, 2002)). ENLACE uses two algorithms to detect cheating and results are invalidated when it happens.<sup>8</sup> There is no evidence of test scores invalidation by the Ministry of Education of Colima to any PAE school during 2010, 2011 and 2012. In addition, the percentage of students who did not take ENLACE in 2010 and 2011 was equal to non-PAE schools, which suggests that PAE schools did not try to manipulate test scores by choosing the students who took the test. Second, student mobility across schools might have affected the test scores after the PAE schools were identified. But such mobility in Mexico is very difficult without a strong reason such as the geographic reallocation of

> 8Algorithms have also been used in the US to detect cheating, see Jacob and Levitt (2003) for an example using data from Chicago public schools.

15
the student’s family. In addition, it would have been more likely that the best students would have moved out of PAE schools, which, if anything, would suggest the impact of the program could have been stronger than suggested by this study. Third, school directors could have changed their managerial practices in PAE schools. There is some evidence, from self-reported surveys, that directors in PAE schools improve monitoring of teacher’s attendance and punctuality, visited classrooms more often, and had meetings to discuss learning outcomes between 2009 and 2010. Finally, principals and teachers may have focused on teaching to the test. Curriculum and ENLACE are linked by design and ENLACE was used to show the weakness areas in the classrooms of the low-performing schools. With the existing information, we cannot rule out the possibility that teachers used this information to teach the subjects that were more closely connected to the test (math and Spanish).

# **4.3 Heterogeneous effects**

The average positive and significant effect of PAE could hide important heterogeneous impacts within schools. The program could have a differentiated effect among boys and girls or among students with adverse initial conditions versus relatively well-off students. Between 2009 and 2010, the student-level dispersion of ENLACE results in PAE schools experienced a larger increase vis-a-vis what was observed for the state as a whole. For instance, between 2009 and 2010, the standard deviation in math test scores among all schools in Colima increased by 4 ENLACE points while the increase was equal to 10 points among PAE schools.

To explore possible heterogeneous effects of the program, separate specifications are estimated for boys and girls and for students identified as having an age-grade distortion and those who do not. Figure 8 shows PAE’s learning outcome effects in math and Spanish, differentiated by gender using the two alternative methodologies, DID and RD. The results show that although the program had some heterogeneous impacts across boys and girls with boys experiencing larger improvements in math and girls in Spanish, we cannot reject the null hypothesis of equality of coefficients between gender-specific equations.

Students with an age-grade distortion in Mexico are largely the outcome of late enrollment in the education system. As shown by Manacorda (2012) age-grade distortions have significant and long-lasting negative effects on learning outcomes. In 2009, more than 17 percent of the students in Colima’s public primary schools had an age-grade distortion and close to 40 percent of them fell in the “insufficient” level in ENLACE compared to 20 percent among students without an age-grade distortion. Figure (9) shows PAE’s learning outcome effects in math and Spanish, for students with an age-grade distortion and those who are on the right grade based on their age, using the two alternative methodologies, 16
DID and RD. The results show that PAE had a positive, though not statistically significant, effect on math and Spanish learning outcomes among students with an age-grade distortion while the effect was positive and significant among students without a distortion. This has two important implications: (1) the improvements in average learning outcomes brought about by PAE were not the outcome of an increase in results among the better-off students while reducing learning outcomes among the relatively worse-off, and (2) students with relatively better starting conditions seem to benefit more from the changes introduced by a low-stakes accountability and diagnosis interventions such PAE. Arguably, improving learning outcomes among students with more challenging starting conditions requires more comprehensive interventions, combining PAE’s diagnosis with specialized pedagogical interventions.

# **5 Conclusions**

In 2009, the state of Colima identified 108 public primary schools that had obtained the lowest learning outcomes as measured by the national standardized student assessment, ENLACE. In early February 2010, the state governor announced the “performance status” of selected schools: schools which performed below an arbitrary cut-off were automatically enrolled in a mandatory school improvement program known as PAE. The program, however, was discontinued during the 2011-2012 school year.

Following two alternative strategies to identify the effects of PAE on learning outcomes, a difference-in-difference and a regression discontinuity design, the paper shows that PAE increased test scores by 0.12 standard deviations only a few months after program launch. The size of the effect aligns with other studies evaluating the impact of low-stakes accountability interventions on test scores. Although this effect remains one and, to a lesser extent, two years after program implementation, our results show no additional impact attributable to the implementation of _pedagogical interventions_ or interventions intended to change major inputs in the learning production function as oppose to a marginal change in effort. The effects are homogeneous across boys and girls; however, learning outcomes among students with disadvantaged initial conditions, proxied by age-grade distortion, improved only marginally (statistically not different from zero) as a result of the intervention.

The fact that the PAE program was halted after only 18 months of implementation suggests that the main intervention of the program was circumscribed to the public announcement made by state authorities, followed by detailed information provided to the schools about the test scores of their students, the activities connected to the design of a school improvement plan, and close support provided by the program’s technical advisers.

17
Activities during the period of preparation of the school improvement plan included the notification to schools that they were low-performing, a diagnosis based on test score results, identification of weaknesses within subject areas evaluated, a discussion between the school director and teachers on how to address the challenges, and the setting of clear goals regarding learning outcomes. In other words, it was the information that was publicly announced apprising directors in PAE schools of their relatively poor performance. While this information was public already, the announcement by state authorities triggered an accountability effect. The diagnostic feedback that came about through the design of the school improvement plan gave the schools the tools and knowledge they needed to take action and set goals themselves. Therefore, it is plausible that the public announcement itself allowed school to make small but significant learning gains using the results of the diagnostic feedback they received.

The results of this analysis indicate that full and wide dissemination of information detailing school quality is critically important. When students, teachers and parents in a school know that their scores are low, and this triggers a process of self-evaluation and analysis, the process itself may lead to an improvement in learning outcomes. Although there was no “shaming” for PAE schools in Colima, there may be an intrinsic motivational impact connected to the ranking of a school relative to others compounded by the compensatory nature of the program and the co-responsibility of state authorities in the challenge of improving learning outcomes. According to this analysis, it is not the inputs made available by PAE that led to improvements. Rather, it was the signaling value of the program – as well as the associated diagnosis and networking opportunities with other – school officials and advisers which resulted in rising test scores. Moreover, unlike the high-stakes accountability interventions sometimes leading school closures in the United States, or the sacking of school directors in England, or the lead with your feet school choice in the Netherlands, the policy (and the _de facto_ events) in Colima bore no punitive actions against schools or school directors.

While the PAE program in Colima was surprisingly and frustratingly short-lived, its premature termination serves to highlight a largely unrecognized phenomenon in education: acknowledgment is, in some ways, virtually tantamount to improvement. After all, if you really understand the problem, effective solutions come much easier. If you do not understand the problem, no amount of “problem-solving” can be expected to work. One may still legitimately wonder why schools did not improve before the PAE program given that the same information was already disclosed publicly. Perhaps the information was not well understood or disseminated, or beleaguered school leaders in poorly performing schools could not, without the right logistical support and networking, begin to proactively use the results from the standardized test to trigger a discussion and design a school improvement plan. These are all areas of future research. It remains refreshing, however, that the use of 18