metadata: authors: 0: author: Lopez-Acevedo,Gladys C.
count: Mexico
docna: 0: docna: Learning outcomes and school
            cost-effectiveness in Mexico : the PARE program
docty: Policy Research Working Paper
subtopic: Primary Education,Educational Sciences,Educational Institutions & Facilities,Educational Populations,Education for Development (superceded),Educational Policy and Planning - Textbook,Effective Schools and Teachers,Education For All
teratopic: Gender
abstracts: cdata!: Past research often attributed most
            differences in student learning to socioeconomic factors,
            implying that the potential for direct educational
            interventions to reduce learning inequality was limited. The
            author shows that learning achievement can be improved
            through appropriately designed and reasonably
            well-implemented interventions. The author studies the
            impact of the Programa para Abatir el Rezago Educativo
            (PARE), a program designed to improve the quality and
            efficiency of primary education in four Mexican states by
            improving school resources. The PARE program increased
            learning achievement in rural and native schools, where
            students had typically not performed as well as other
            students (in Spanish). Not only did students' cognitive
            abilities improve under the PARE program, but the
            probability of their continuing in school improved. In rural
            areas where the PARE design was fully implemented, test
            scores for the average student increased considerably. A 30
            percent deficit in test scores among rural students could be
            overcome by roughly doubling the resources allocated per student.
geo_reg_key: 119228,517191
repnme: repnme: Learning outcomes and school
            cost-effectiveness in Mexico : the PARE program
display_title: Learning outcomes and school
            cost-effectiveness in Mexico : the PARE program
listing_relative_url: /research/1999/05/441000/learning-outcomes-school-cost-effectiveness-mexico-pare-program
topicv3: Teachers Management m1326119 1523,Primary Education m1326139 480,Textbooks m1326802 2129,Education m1327048 470,Teacher Performance m1327509 1531
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Gender
origu: Central America Countries CMU (LCC2C)
no_of_pages: 27
author_keys: 0: author_key: 428192
model_extractions: POLICY RESEARCHI WORKING PAPER 2128

# **Summary** findings

Past research often attributed most differences in student learning to socioeconomic factors, implying that the potential for direct educational interventions to reduce learning inequality was limited.

Acevedo shows that learning achievement can be improved through appropriately designed and reasonably well-implemented interventions.

She studies the impact of the Programa para Abatir el Rezago Educativo (PARE), a program designed to improve the quality and efficiency of primary education in four Mexican states by improving school resources.

The PARE program increased learning achievement in rural and native schools, where students had typically not performed as well as other students (in Spanish). Not only did students' cognitive abilities improve under the PARE program, but the probability of their continuing in school improved.

In rural areas where the PARE design was fully implemented, test scores for the average student increased considerably. A 30 percent deficit in test scores among rural students could be overcome by roughly doubling the resources allocated per student.

- - This paper a product of the Mexico Country Management Unit, Latin America and the Caribbean Region is part of a larger effort in the region to understand the impact of program intervention in Mexico. Copies of the paper are available free from the World Bank, 1818 H Street NW, Washington, DC 20433. Please contact Michael Geller, room 14-142, telephone 202-45 _8-5155,_ fax 202-522-2093, Internet address mgeller@worldbank.org. Policy ResearchWorking Papers are also posted on the Web at http://www.worldbank.org/html/dec/Publications/Workpapers/home.html. The author may be contacted at gacevedo@worldbank.org. May 1999. (23 pages)

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas ahout development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represent._ Produced by the Policy Research Dissemination Center
# Learning Outcomes and School Cost-Effectiveness in Mexico: The Pare Program*

## **Gladys Lopez Acevedo**

*Gladys Lopez is an Economist, the World Bank (Mexico Department) and an Economics Professor at CIDE and at ITAM.

This paper was prepared with research assistance from Mireya Pasillas.

1
**Learning Outcomes and School Cost-Effectiveness in Mexico: The Pare Program**

# Gladys Lopez-Acevedo

# **1. Introduction**

To understand better the qualitative dimension of basic education, it is necessary to analyze student learning outcomes and school effectiveness. What factors influence them? How responsive is student learning to these factors? What impact can learning improvement interventions have? This paper attempts to address some of these questions.

Empirical studies of student learning achievement in Mexico are scarce. Interest, however, regarding its determinants and the impact of interventions to improve it is increasing. In January 1995, the Ministry of Education presented the Programa de Desarrollo Educativo 1995-2000 (PED), which contains a series of targets and general guidelines in order to improve the coverage, efficiency and equity of the Mexican educational system. In fact, the PED recognizes the importance of research and evaluation in its strategy to improve quality of education. In view of this policy, the Ministry of Education has collected databases that can be useful for this purpose. These include Carrera Magisterial, PARE, and TIMSS (Third International Mathematics and Science Study) databases among others.

Due to data constraints, this paper is unable to do a comprehensive and in-depth analysis of the learning achievement issues in Mexico. The following analysis, therefore, should be regarded as an exploratory rather than as a conclusive study. Available data are used to highlight certain ideas about learning improvement interventions. As mentioned, there has been very little study in Mexico that examined this issue. There are, however, many international studies that looked at this question. An excellent summary of this literature can be found in Fuller and Clarke (1994), and Hanushek (1995).

The early studies on learning outcomes showed that the student's socio-economic and cultural background predominantly determines differences in test scores. These led to the conclusion that there was little that government can do by way of direct educational policy and government interventions to improve learning outcomes. More recent results and experience, however, indicate that school factors do matter and that they can play a more critical role than previously thought. Moreover, "education production function" studies indicate that the magnitude of production inputs varies substantially. Some inputs have larger marginal effects than others do and, in some places, the effects of some of the factors are not statistically significantly different from zero, while in others the same factors have shown substantial impact.

Table 1 summarizes the various educational inputs that have been empirically analyzed, the number of studies reviewed and the "confirmation percentage" for each of 2
the inputs. Confirmation percentage is defined as the proportion of the reviewed studies showing positive and significant relationship between the specific input and test scores. At the primary level, it is clear that class instructional time, school library, textbooks, and class frequency of homework have the highest confirmation rates at 73.1 - 88.9 percent. On the other hand, teacher's salary level and school teacher/pupil ratio have the lowest confirmation rates at 36.4 and 34.6 percent. More recent studies also tend to stress the effectiveness of improving of physical facilities. Relating the cost of these inputs to their marginal effects on test scores, available estimates further show that in fact textbooks and other educational materials along with improvement of physical facilities have much higher cost-effectiveness than increased teacher salary, years of experience and teacher/pupil ratio.

|**Table 1. Confirmation Percentages of Various Educational Inputs Sorted by Direct**|
|---|
|**Importance to Teacher Utility**|

||Number of<br>studies|Positive and<br>sianificant relation|Confirmation<br>Percentaae|
|---|---|---|---|
|**Primary**Schools:||||
|Teacher's salary level|11|4|36.4|
|School teacher pupil ratio|26|9|34.6|
|Teacher's years of schooling|18|9|50.0|
|Teachees experence|23|13|56.5|
|Class instructional time|17|15|88.2|
|Class frequency of homework|<br>11|9|81.8|
|School library|18|16|88.9|
|School textbooks|26|19|73.1|
|**Secondary Schools:**||||
|Teachers salary level|11|2|18.2|
|School teacher pupil ratio|22|2|9.1|
|Teacher's expenence|12|1|8.3|
|Class instructional time|16|12|75.0|
|School textbooks|13|7|53.8|

Source: Fuller and Clarke (1994).

Several lessons might be drawn from these studies. First, given the abovementioned differential effects, it is not surprising that differences in aggregate education' budget does not appear to have a tight association with learning outcomes. It all depends on how budgets are allocated and used. Second, in the absence of local information about the relative effectiveness of inputs, improving availability of text books, workbooks, educational materials, school library, and physical facilities would be a prudent choice over other inputs such as increasing teacher/student ratio, teacher salary, and experience especially if schools have a shortage of the previous type of inputs. Nevertheless, in view of the findings that the relative impact and cost of particular inputs depends on the local conditions of schools and their student, it is important to collect local information about the issue.

3
Beyond the above issues, there is a need to understand the structures and processes needed to establish a motivating and enabling environment to ensure that highly cost-effective inputs and interventions are indeed chosen. It becomes also extremely important to think carefully about the appropriate program design and implementation strategy.

This paper presents some empirical analyses of learning outcomes based on local data and experience regarding the impact of Programa para Abatir el Rezago Educativo (PARE). The paper is divided as follows. The next section describes the PARE Program and the database. Section 3 assesses the impact of the PARE program on learning and achievement. Section 4 evaluates the cost-effectiveness of the PARE program. The final section presents concluding remarks.

# **2. PARE background**

Programa para Abatir el Rezago Educativo (PARE), 1992-1997. The objective of the program was to assist the Government of Mexico in improving the quality and efficiency of primary education, focusing on four Mexican states (Oaxaca, Guerrero, Chiapas and Hidalgo) with the highest incidence of poverty and low education indicators. These objectives, considered as being of the highest priority within the Government's Education Modernization Program, would be achieved through; (i) reducing the high repetition and dropout rates; (ii) raising the level of cognitive achievement of children, and (iii) strengthening management of the primary education system, including program design and implementation, monitoring and evaluation of the system. The program consisted of giving schools additional resources (components) like libraries, better distribution of textbooks, academic material, training aid to teachers and principals, increased in official supervision of teachers and construction and repair of schools.

From its inception its performance was monitored through statistical comparisons between the target, or _experimental,_ population (schools in the states of Chiapas, Guerrero, Hidalgo and Oaxaca) and a _control_ group formed by students in comparable schools in the state of Michoacan which falls outside the scope of the program. Special surveys were conducted yearly between 1992 and 1995. In addition, all students were given standardized achievement tests in Spanish and mathematics. PARE also provided the resources to evaluate the success of this program. To this end, two studies were - conducted for two different research institutions. One study was made by the C.E.E, mainly through quantitative variables on school, parents, community, teachers, inputs, supervisors, socioecomic and academic background, and the other by the D.I.E (Departamento de Investigaciones Educativas), through qualitative variables. These databases were developed to evaluate the effects of PARE (Programa para Abatir el Rezago Educativo) on student achievement.

During the program several test on Mathematics and Spanish were applied to the students in three consecutive years, when they were in fourth, fifth and sixth grades. The scores of these tests give the outcome or output variables and at the same time allow us to use a value-added estimation. The C.E.E staff also evaluated school directives and school 4
characteristics. Students' parents and teachers answered a survey at the same time. This information was needed to control for teacher and socioeconomic characteristics.

The C.E.E sample consists of students from 198 schools randomly chosen from four different types of schools: Urban, RURAL, NATIVE and CONAFE from five different states.' The participation of each school type, relative to the total is shown in Table 2.2 Table **2. Number of schools by type and state, 1992**

|State|**Urban**|**Rural**|**Native**|**CONAFE**|**Total**|
|---|---|---|---|---|---|
|Chiapas|6|13|14|5|38|
|Guerrero|4|14|12|4|34|
|Hidalgo|3|11|12|8|34|
|Oaxaca|7|17|15|12|51|
|Michoacan|7|15|10|9|41|
|Total|27|71|64|44|198|

Source: PARE's database.

# **3.** Impact of the PARE **program on learning and achievement**

# 3.1 Control and the experimental groups

The literature generated by the PARE points toward a mixed conclusion about the impact of the program.<sup>3</sup> This was partly due to incomplete and faulty implementation, especially in urban areas. By design, the program intended to provide a number of simultaneous actions (components), which together would impact on educational outcomes. For pedagogical reasons the total was to be greater than the sum of the parts. The actions were to affect the behavior of students, parents, teachers, principals and supervisors; they were to provide the target schools with supplies, didactic materials and 1 CONAFE stands for Consejo Nacional de Fomento Educativo.

2 The Native school refers to schools offering services to populations which mother tongue is not the Spanish.

3 The PARE program has generated a voluminous literature produced mainly by the _Direcccion General de Evaluacion_ of the _Secretaria de Educacion Publica_ and by the _Centro de Estudios Educativos A. C._ (CEE). The CEE was chosen by the executing agency of the PARE program (the _Consejo Nacional de Fomento Educativo,_ CONAFE) to monitor and evaluate the program. Its conclusions were summarised in the document "Determinacion del Impacto del PARE en el Aprovechamiento y la Retencion Escolares," Tercer Infonne, Tomo IV, Mexico, D.F., March 1996. After an extensive analysis of the data the report concludes (my translation) on page 21: "... the variable PARE [a dichotomous variable identifying schools which had access to the program] had a significant impact in only two of the estimated equations. They are, first, the equation referring to performance in mathematics in urban schools of the states' capitals; second, the equation for performance in Spanish in rural schools closer to the states' capitals. ... only for schools in these two sub-samples did students achieve performance levels greater than those in comparable schools which remained outside the PARE program." _5_
physical infrastructure. In fact, however, only a sub-set of schools benefited systematically from all actions what will be called from now on components 1358.

To assess the probable impact of the PARE program we consider a number of experiments based on the following question: What would have been the program's historical performance if it had been implemented as envisaged without faults or delays. We construct counterfactual experiments based only on those schools, which received all of the main components of the program. Before going into the analysis, it's important to mention that the information available posed important constraints for building a panel data set.

Table 3 shows the distribution of students by school type in the sample. Our analysis will focus on schools located in rural and native communities, the two most disadvantaged groups in the population with the lowest educational attainment, poorest test performance and highest incidence of school desertion. At the margin, the supplemental actions provided by the program should have the greatest impact amongst this population. Table 4 shows the resulting samples for analysis considering that, for the reasons already noted, we concentrate our attention on a sub-set of these schools -- those which benefited integrally from the program.

**Table 3. Distribution of students by school type,** 1992.

|**CWapa**|**s Guetrem**|**Fidalgo C**|**Oxam**|**SA,**<br>|**cta:**|IVichoacan:|T|
|---|---|---|---|---|---|---|---|
|LIban<br>396|<br>107|257|357||1,119|<br>361|1,480|
|**Rual**<br>200|<br>202|175|239||816|<br>208|1,024|
|Nafive<br>197|<br>114|122|259||692|<br>205|897|
|Conru*<br>19|<br>11|29|59||118|<br>Z7|145|
|**TOaW**<br>814|<br>434|583|914||2,745|<br>801|3,546|
|Source: PARE's databa<br>**T**<br>**Native &**<br>**Rural.**<br>**With**|se<br>**able 4. St**<br>**Native &**<br>**Rural. With**<br>**comp. 1358**|**udents inclu**<br>**Sub-total**<br> <br>**included in**<br>|**ded i**<br>**Nati**<br>**Rural.**<br>**so**|**n the**<br>**ve &**<br>**With**<br>**me**|<br>**analysis**<br>**Urban**<br>**Commu**|<br> <br>**, 1992.**<br>**&**<br>Sub-total<br>excluded<br>**nity**<br>**from the**|<br>**TOTAL**|
|**comp. 1358**|**and other**|**the analysis**|**compo**|**nente**|**School**|**s**<br>**analysis**||
|Experimental<br>585|624|<br>1,209||299|1,|237<br>1,53|6<br>2,745|
|Control<br>0||0<br>413||0||388<br>38|8<br>801|
|**TOTAL**<br>585<br>Of which: Native<br>Rural|624|<br>1,622<br>769<br>853||299|1,|625<br>1,92|4<br>3,546|

Source: PARE's database.

6
We measure performance by the student's score obtained in the tests applied at the beginning of the 4th grade - before the program began- and at the conclusion of the 6th grade, when the program was already in its third year of implementation. The tests were designed and applied by the _Direccion General de Evaluacion_ (DGE) of the _Secretaria de Educacion._ Notice that in the opinion of both the DGE and of the CEE, which conducted the impact evaluation of the program, the Spanish test provides a superior metric. Students' performance in mathematics was very low.

Measured by their scores in Spanish, the performance of students in the experimental group of schools is significantly better in both the rural and native subsamples. As shown in Table 5, before the program, students in native schools in the experimental group were markedly disadvantaged with respect to their peers in the control group. The program eliminated this difference. Students in rural schools were undifferentiated before the program; with the program, those in the experimental group showed significantly higher scores. The percentage change in performance is, on average, three times as large for students in the experimental group. However, in urban areas a retrocession in student's performance was observed probably because bad implementation or the wrong components.

**Table 5. Student's change in performance, 1994.**

||**Befo**|**re (1992)**|**After (1994)**|**Di**|**fference**|
|---|---|---|---|---|---|
||**Students**|**Average test**|**Students Average s core**|**Total**|Percentage|
|Native||||||
|Experimenl|564|14.6|356<br>29.1|13.9|95.3|
|Control|205|23.2|125<br>26.8|4.1|17.7|
|Total - t/test|769|16.9|481<br>28.5|11.4|67.3|
|**Rural**||||||
|Experimernal|645|20.7|421<br>32.9|11.6|56.0|
|Control|208|20.1|128<br>29.7|8.2|40.6|
|Total -ttest|853|20.5|549<br>32.1|10.8|52.5|
|**Urban**||||||
|Experimental|337|26.9|238<br>39.7|12.0|44.5|
|Controi|361|26.9|221<br>44.3|15.9|59.3|
|Total - tttest|698|26.9|459<br>41.9|13.9|51.6|

Source: Own calculations based on PARE's database. Note: Difference respect to control group.

7
3.2 Regression analysis on the impact of the PARE program on learning and achievement In this subsection, we assess the impact of the PARE intervention on students' scores controlling for supply and demand indicators. The results are shown in Tables 6 through 8.

The indicators, constructed through principal components analysis, include:

- Family's cultural capital: index based on parents' schooling, reading habits, television and radio programs listened, and number of books at home;

- Teacher performance: index based on teacher's attendance and other practices;

- Quality of school director: variable indicating favorable school conditions for teaching and learning, such as qualification of principal, his knowledge update, and the distribution students in the classroom.

- Supervision quality: a composite indicator based on frequency of supervisor visits, duration, occupations of people interviewed, and themes discussed; and

- Parents' participation: a measure of parents' attitudes to teachers' attendance, participation in school activities, and relevance of parents' school association.

**Table 6. Student's change in performance, 1992 and 1994.**

||**Native**|<br>**Rural**||
|---|---|---|---|
||**Beta coefficient**|**tvle**<br>**Beta coefficient**<br>**t-valuet-au**|**t-value**|
|Control|0.245348|4.698a<br>0.11465|2.695a|
|Teachefs Performance 6th grade|-0.002794|-0.060<br>0.074814|1.691c|
|Teachers Performance 5th grade|-0.004594|-0.102<br>0.107404|2.485a|
|Director's Quality|0.171017|3.709a<br>0.138362|3.040a|
|Supervision Quality|0.121867|2.302b<br>0.013111|0.283|
|Parents' Participation|0.072675|1.565c<br>-0.133568|-3.048a|
|Child'spartAcademic Record|0.043542|0.984<br>0.061537|1.441d|
|R2-adjusted||0.12097|0.06234|
|F||10.437a|6.205a|
|N|480|548||
|Students self-esteem at 5th grade|-0.088019|-2.032b<br>-0.044318|-1.039|
|Availability & quality of urban infrstructure|-0.166759|-3.120a<br>0.006836|0.153|
|Memorandum item:||||
|Maximum total contribution of PARE program|0.530844|0.448341||

a - Significant atthe 1% level ormore b - Significant at the 5% level or more

- c - Significant at the 10-% level or more

- d - Significant at the 20% level or more

Dependent vanable: Difference in normalized test scores between 6th and 4th grade 8
Source: Own calculations based on PARE's database.

Table 6 shows a simple ordinary linear square model that captures only about 6% of the variance in the difference of scores (between 4th and 6th grades) amongst students in rural schools and 12% amongst students in native schools. No doubt this reflects an inadequate specification of the model be it in its functional form or inclusion of relevant explanatory factors. To the extent that the measured test scores fail to capture the true level of performance in the sample, much of the influence of variables such as parental background, the quality of teaching, etc., is lost in the model. The point to note, however, is that, even so, the explanatory variables behave as expected.<sup>4</sup> More importantly, the coefficient of the experimental variable is large and significant. The PARE program has a large positive impact on student achievement in this counterfactual experiment by all means in the scenarios and specifications.

The impact is larger for the native schools, a result that is consistent with the orientation of the program. As reported in Table 6, the marginal contribution of each explanatory variable is measured in terms of standard deviations of the dependent variable; i.e., of the percentage change in performance between 4th and 6th grades. This is in order to control for possible demand driven effects and hence simplify the analysis. For the average student at native schools, attendance at a school fully served by the program would, on average, increase the percentage change by 25%. The comparable percentage change for students attending rural schools is half as large. The variables- of school ".supply" (the performance of teachers, principals and supervisors) are partly an _outcome_ of the program. Thus, the program, at its _maximum_ effect estimated with the results of Table 6, could increase the performance of the average student by one-half of the standard deviation of the percentage change in test scores for the respective sub-ample.

It should be noted that the variables measuring the characteristics of students, parents, school personnel and facilities are all numerical indices constructed by C.E.E analysts. Some indices aggregate answers to as many as a dozen questions in the original survey. The model in Table 6 is a simple, parsimonious representation. In particular, it could be argued that if the characteristics of the demand (family and community background, parental attitude towards and involvement in schooling, academic history, self-esteem, etc.) were adequately measured, the additional effect of the PARE program would be smaller, even insignificant. Alternatively, if the characteristics of the supply 4 Three observations may be pertinent. First, for students attending native schools it seems that selfesteem, measured at 5th grade and residence in a community with greater access to public services is _negatively_ correlated with performance. One plausible explanation is due to the conflictual character of native education: Students that are positively self-selected may have a greater resentment in attending special schools. Second, and for the same group, while the performance of teachers does not seem to alter significantly the perfornance of students, the performance of principals and supervisors does. This result may be due to the generally poor quality of teaching in native schools. Finally, it is puzzling to note that, in the nrral sub-sample, parental involvement _diminishes_ students' performance. One possible reason for this is the possibility that parental involvement increases as the quality of the school diminishes. Parents act only when the problems are large and apparent.

9
(teachers, principals and supervisors background, performance, attitudes, assiduity, pay, etc., as well characteristics of the school infrastructure and availability of textbooks, supplies, etc.) were captured more precisely, the impact of the program could be larger. The data allows us to do better than the simple model of Table 6; and to make use of the available information without introducing damaging multicollinearity in the results we constructed two sets of principal (orthogonal) components measuring respectively the characteristics of the demand and supply of schooling.

Table 7 shows the results of the model built on this more complex structure captured through the two principal components. _The results are very similar to those of Table 6. In fact, the impact of the program is greater and more significant. The factor capturing the conditions of supply is also significant and large, especially in the case of schools serving native communities._

**Table** 7. Student's change in performance, 1994

||**Nativ**|**e**|Rural||
|---|---|---|---|---|
||Beta coefficient|<br>t-score|Beta<br>coefficient|t-score|
|Control|0.273609|<br>6.210a|<br>0.12721|4 3.000a|
|Factor - Characteristics<br>ofcommunity<br>& family|-0.009075|<br>-0.205|<br>-0.16103|3 -3.815a|
|Factor - Characteristics<br>ofschool & system|0.201875|<br>4.664a|<br>0.07444|9 1,754c|
|R2-adjusted||0.12376||0.035|
|F||23.599a||7.713a|
|N|480||548||
|Memorandum<br>item:|||||
|Maximum total contribution<br>of PARE program|<br>0.475484||0.20166|3|
|a - Significant at the 1% level or more|||||
|b -Significant at the 5% level or more<br>c - Significant<br>at the 10%/o<br>level or more|||||
|d - Significant<br>at the 20% level or more|||||

Dependent variable: Difference in normalized test scores between 6th and 4th grade. Source: Own calculations based on PARE's database.

An objection may be raised, nonetheless, about the measure of performance. What if small differences in test score are very imperfect measures of relative capabilities and/or achievements? To try to get around this issue, we perform a final experiment on the test scores. We stratify the samples in two sub-samples each: those of students with performance above and below their respective medians. These results are shown in Table 8. Once again the estimates are consistent. The program has a positive and significant impact, and especially so for the native population.

10
**Table 8. Student's change in performance, 1994.** Dependent variable: Probability of testing above the median in **6h** grade.

||**Coefficient**|**Std. Error**|**t-Statistic**|**Prob.**|
|---|---|---|---|---|
|**Native schools**|||||
|Constant|-0.991|0.217|-4.572|0.0%|
|Control|1.272|0.246|5.162|0.0%|
|Factor - Characteristics of community & family|0.054|0.103|0.528|59.8%|
|Factor - Characteristics of school & system|0.630|0.104|6.056|0.0%|
|N||481|||
|Log likelihood||-295.562|||
|F-statistic|15.024|||0.0%|
|Chi-square|60.095|||0.0%|
|Obs with Dep= 1|237||||
|Obs with Dep=0|244||||
|Ex-ante probability|49%||||
|Estimated probability (at means)|49%||||
|Estimated probability without PARE (control)|27%||||
|PARE contribution - percentage gain probability||45%|||
|**Rural schools**|||||
|Constant|-0.396|0.183|-2.161|3.1%|
|Control|0.495|0.209|2.372|1.8%|
|Factor - Characteristics of community & family|-0.233|0.086|-2.713|0.7%|
|Factor - Characteristics of school & system|0.107|0.083|1.279|20.1%|
|N||549|||
|Log likelihood||-374.073|||
|F-statistic|3.112|||1.5%|
|Chi-square|12.448|||1.4%|
|Obs with Dep= 1|271||||
|Obs with Dep=0|278||||
|Ex-ante probability|49%||||
|Estimated probability (at means)|49%||||
|Estimated probability without PARE (control)|40%||||
|PARE contribution - percentage gain probability||19%|||
|Source: Own calculations.|||||

11
**Table 8a. Probability of being in school in the 6th grade, 1994 (Being at school in the 4th grade)**

|**Variable**<br>**Coefficient**<br>**St**|**d. Error**<br>**t-Statistic**|Prob.|
|---|---|---|
|**Native**<br>**schools**|||
|Constant<br>0.499|0.148<br>3.379|0.001|
|Control<br>0.115|0.174<br>0.660|_0.510_|
|Factor-<br>Characteristics<br>ofcommunity<br>& family<br>0.125|0.078<br>1.599|0.110|
|Factor<br>- Characteristics<br>of school<br>& system<br>-0.067|0.076<br>-0.876|0.381|
|N<br>769|||
|Log likelihood<br>-500.106|||
|F-statistic<br>15.597||0.000|
|Chi-square<br>62.386||0.000|
|Obs with Dep=l<br>493|||
|Obs with Dep=0<br>276|||
|Ex-ante<br>probability<br>64%|||
|Estimated<br>probability<br>(at means)<br>64%|||
|Estimated<br>probability<br>without<br>PARE<br>(control)<br>62%|||
|PARE<br>contribution<br>- percentage<br>gain probability|3%||
|**Rural**<br>**schools**|||
|Constant<br>0.496|0.144<br>3.441|0.1%|
|Control<br>0.271|0.168<br>1.613|10.7%|
|Factor<br>- Characteristics<br>of community<br>& family<br>0.184|0.076<br>2.419|1.6%|
|Factor-<br>Characteristics<br>of school<br>& system<br>0.121|0.075<br>1.617|10.6%|
|N<br>825|||
|Log likelihood<br>-519.618|||
|F-statistic<br>23.752||0.0%|
|Chi-square<br>95.010||0.0%|
|Obs with Dep=1<br>549|||
|Obs with Dep=0<br>276|||
|Ex-ante<br>probability<br>67%|||
|Estimated<br>probability<br>(at means)<br>67%|||
|Estimated<br>probability<br>without<br>PARE<br>(control)<br>62%|||
|PARE<br>contribution<br>- percentage<br>gain probability|7%||

Source: Own calculations.

Table 9 summarizes the results on test scores. The PARE program - when adequate and fully implemented - could cause an increase in performance for the average student in the range of 19 to 38% amongst rural students. For native students, the percentage change could be much larger, anywhere from 45 to 90%. If consideration is taken of the factors affecting supply, such as the performance of teachers, principals and supervisors, on the plausible assumption that this performance is in part a product of the program, the total impact could be even larger.

12
**Table 9. Marginal contribution of belonging to the experimental group,** 1994.

||**Group**|**Mean of dependent**<br>**variable**|<br>**Unit**|**Estimated coefficient**<br>**Exnerimental**|**Marginal contriibtion:**<br>**Exneximental**|
|---|---|---|---|---|---|
|Table 6|Rural|10.778|Gain in scores|4.043|0.375|
||Native|11.356|Gain in scores|9.998|0.880|
|Table 7|Rural|10.778|Gain in scores|3.644|0.338|
||Native|11.356|Gaininscores|10.259|0.903|
|Table 8.|Rural|0.494|Probability|0.495|0.190|
||Native|0.493|Probability|1.272|0.450|
|**Table 8a**|Rural|0.670|Probability|0.271|0.070|
||Native|0.640|Probability|0.115|0.030|

Source: Own calculations.

Note: For Tables 6 and 7, the percentage gained to the mean. For Table 8, the percentage gained to the initial probability of success, estimated at the means of the independent variables.

_Aside from increasing the student's cognitive achievements while at school the PARE program also increases the probability that the student will continue in school._ The two outcomes are probably linked: children who perform better are more motivated to continue and their parents may be more inclined to allow them to continue in school. This is clearly the case for rural students, as shown in Table 10. The probability of school desertion is 20% lower amongst students supported by the program, and the effect is just as large for the broader group of students who benefited from only a partial application of the program. Surprisingly, however, the result does not seem to hold for the native population. One-third of the native students who received the full program from 4th grade onward abandoned the _school before completing_ the 6h grade. Their probability of desertion was 12% _greater_ than that of the comparable control group.

**Table 10. Desertion.**

**Percentage of students who quit school by the end of the 6th grade,** 1994.

||**Complete Pro**|**aram***|**Partial oro**|**cram**|
|---|---|---|---|---|
||**Native**|**Rural**|**Native**|**Rural**|
|Experimental|32.9%|28.4%|36.2%|31.1%|
|Control|29.4%|35.7%|36.0%|38.5%|
|Difference|11.7%|-20.5%|0.7%|-19.2%|
|N|698|809|841|1,006|

Source: Own calculations based on PARE's database.

* Students in school that received all PARE components simultaneously.

This result deserves more analysis. An intriguing possibility is that high-achieving students in native communities move to rural schools where they are immersed in a Spanish-speaking environment. On the other hand, a multivariate analysis (controlling for "<sup>4</sup> supply" and "demand" variables) of the probability that the student was in school in the 6t grade (given that she had been at school in the 4t grade) indicates that the program had a positive impact on both rural _and_ native schools, see Table 8a. The percentage change 13
in probability is small, however, and specially so for the native population (a mere 3 percent increase).

Due to the lack of adequate and sufficient number of instruments we could not sort out the intriguing findings posed by the sign or significance of some of the variables. No doubt, in all the models the experimental variable was significant and positive.

# **4. Cost-effectiveness of the PARE program**

# 4.1 Costs in the PARE program

It is very difficult to estimate the true costs of the PARE program. The program, financed by CONAFE, is not independent of actions taken by SEP in its usual activities of finding and supervising basic education, as explained in Section 2. It could be, for example, that teachers in a school benefiting from the PARE program become more motivated and assiduous simply because they perceive the threat (or reward) of closer supervision by the educational authorities. The costs of the PARE program, as reported by the C.E.E, are shown in Table 11. Expenditure on native schools was nearly 60% higher compared to rural schools and 786% higher respect to urban schools.<sup>5</sup> The largest cost items were infrastructure and materials. Expenditure on teacher training and wage incentives accounted for less than 14 % of total spending.

**Table 11. Per pupil expenditure, 1994.**

|**A**|**lI**schools*||**PARE****||**Cos**|**t**increas|e|
|---|---|---|---|---|---|---|---|
|||Native|Rural|Urban|Native|Rural|Urban|
|Chiapas|1,983|605.7|338.1|<br>210.7|<br>30.5%|17.0%|<br>10.6%|
|Guerrero|2,253|749.1|764.2|<br>62.8|33.2%|33.9%|<br>2.8%|
|Hidalgo|2,143|1,127|636.8|<br>51.0|52.6%|29.7%|<br>2.4%|
|Oaxaca|1,770|624.1|229.7|<br>23.8|35.3%|13.0%|<br>1.3%|
|**Average**|**2,037**|**776.4**|**492.2**|<br>**87.1**|<br>**_38.1%_**|**_24.2%_**|<br>**4.3%**|

Source: PARE's database.

* Unit cost for primary schools in native communities, SEP.

** See Table 12.

> 5 The percentages are obtained as follows: the difference in cost increase between native and rural areas (and native and urban areas) is divided by the cost increase in rural (urban) strata.

14
Table 12. Per pupil costs PARE program, 1994

|**BirIii**|ary|Stares|Training Inr|asucture<br>teadh|ems|**su**<br>Mtrasupriinntd|**pervision**|<br>**is**|**Toal**|
|---|---|---|---|---|---|---|---|---|---|
|Per pupil expenciture<br>- Indgenou**s**chools||||||||||
|Chiapas<br>21.5|<br>3.5|2.0|<br>45.6|215.2|0.0|<br>111.3|47.0|159.5|605.7|
|Guewrero<br>20.2|<br>3.2|<br>4.1|<br>50.1|282.1|45.8|<br>101.1|103.2|139.4|<br>749.1|
|Hdalgo<br>25.2|<br>7.4|3.3|<br>62.0|635.2|123.4|<br>78.3|82.3|109.8|1126.7|
|Oaxaca<br>9.2|<br>4.6|3.5|<br>50.3|279.7|52.8|<br>139.3|29.2|55.6|624.1|
|Averagecost<br>19.0|<br>4.7|3.2|52.0|353.1|55.5|107.5|**65.4**|**116.1**|**776A**|
|Perpupilexpenditure<br>-Rural schools||||||||||
|Chiapas<br>0.0|<br>7.3|3.2|<br>32.7|32.6|21.2|<br>136.4|39.4|65.4|338.1|
|Guerrero<br>0.0|<br>4.8|6.4|40.8|333.0|4.0|<br>97.9|139.9|137.5|764.2|
|Hdalgo<br>0.0|<br>10.6|<br>4.1|58.7|302.4|16.8|<br>93.6|62.4|88.2|636.8|
|Oaxaca<br>0.0|<br>6.1|4.4|42.9|0.0|0.0|<br>94.2|46.5|35.7|229.7|
|Avewagecost<br>0.0|<br>**7.2**|**4.5**|<br>43.8|**167.0**|10.5|105.5|72.1|81.7|**492.2**|
|Per pupil expenciture<br>- Urban schools||||||||||
|Chiapas<br>0.0|<br>5.0|4.0|48.1|0.0|0.0|<br>-|20.1|133.5|210.7|
|Guerrero<br>0.0|<br>1.6|<br>1.0|<br>27.3|0.0|0.0|<br>-|9.8|23.1|62.8|
|Hdalgo<br>0.0|<br>1.4|<br>0.6|28.3|0.0|0.0|<br>-|13.4|7.3|51.0|
|Oaxaca<br>0.0|<br>0.8|0.3|<br>16.4|0.0|0.0|<br>-|5.8|0.6|23.8|
|**Averagecost**<br>**0.0**|<br>2.2|1.5|30.0|0.0|0.0|-|12.3|41.1|87.1|

15
As shown in Table 11 and Table 12, the PARE program increased the average per pupil cost of education by 38% in native schools, by 24% in rural schools and by 4 % in urban schools. A simple comparison between the percentage change in average test scores and the cost of the supplementary pedagogical actions under the PARE program - _for_ the subset of schools that received all of the actions and implemented them accordingly - suggests that the program was well implemented for the native population. Here we observe a 42% in average scores versus the 38% increase in cost, an elasticity of 11% (Table 13). _However, the ratio is negative for the rural and urban population; the increase in cost is greater than the percentage change in performance._ In particular, for urban areas the elasticity was - 445%, which may implied that the implementation of the PARE program was bad in this sector of the population.6

**Table 13. PARE, Program: Cost** Elasticity, 1994

||**Avera,e gain in test score**||**P**|**ercentae**|Increase in|Ratio|
|---|---|---|---|---|---|---|
||**Exs1ereal**<br>**C**|**ontrol Di**|**fference**|**ain**|cost||
|Native|||||||
||13.9025|4.1|9.8|42.3%|<br>38.11%|11.02%|
|Rural|||||||
||11.5746|<br>8.2|3.4|17.0%|<br>24.16%|-29.67%|
|Urban|||||||
||11.9755|15.9|-4.0|-14.7%|<br>4.27%|-445.00%|

Source: Own calculations.

***** With respect to base year - control group; see Table 5.

****** See Table 11 and Table 12.

Instead of using the observed outcomes as reported in Table 13 we could use the simulated outcomes as reported in Table 9. The results are better. Considering the _maximum_ estimated impact for the native population (a _maximum_ percentage change in performance of 90% estimated in Table 6) the benefit/cost elasticity is 137:100. The equivalent ratio for the rural population (with a maximum change in performance of 38% estimated in Table 7) is 58:100.

# 4.1 Cost-effectiveness estimates in the PARE program

The previous analysis looks at the impact of PARE interventions as it was implemented on average, without limiting the assessment to cases where the program wag fully implemented as envisioned. Specifically, the present section seeks to directly relate the monetary value of the PARE assistance actually received by the schools regardless of the original amount originally planned for them.

As explained earlier, ordinary least squares regression was initially used to estimate the relationship. _However, the results shaw a "perverse" negative relationship between PARE expenditure per student and learning outcomes, strongly indicating that schools_

> **6** Regression analysis was used to test for this hypothesis controlling for placement effects. The results support the initial hypothesis.

16
_that were lagging behind in learning achievement were systematically being targetedfor more assistance._ Consequently, a two-stage least squares methodology was used, where the monetary value of PARE assistance per student was modeled as a function of school characteristics and a dummy variable for being in the experimental group or not. This dummy variable is used to identify the learning achievement equation.

The results, which are presented in Table 14, reveal that on average PARE assistance has had a significant positive effect on learning outcome in Spanish. Moreover, they show significant positive fixed effects for the quality of school management, supervision and teachers. The surprising result is that parental participation has a significant negative coefficient. Considering the importance that education reformers attached to this factor, further analysis is called for by this unexpected finding. A possible explanation for this "perverse" finding is that disadvantaged schools are forced to mobilize parents for additional resources. Or, it might be that when children perform badly, their parents take a more proactive role in student learning.

The elasticity estimates appear reasonable. There are several things worth noting here. First, a 10 percent improvement in staff performance and quality as well as the family's cultural capital is associated with about one to two percent increase in test score. Second, a ten percent increase in per student expenditure that is devoted to finance to PARE program activities would likely raise Spanish learning achievement by about 3.3 percent. This is roughly half the above-mentioned full implementation cost-effectiveness estimate of PARE. Third, being in rural area reduces learning achievement by 31 percent. If a student is in a rural and native school, his score is about 75 percent less than that of others.

17
**Table 14. Determinants of Sixth Grade Spanish Test Score: PARE, 1994.**

||**Coefficient**|**t-value**|**Means**|**Std. Dev**|**Elasticity**|**Average**<br>**Spending**<br>**Elasticity**|
|---|---|---|---|---|---|---|
|Child's part Academic Record|-0.1400|-1.6850|70.8100|19.6400|-0.3470||
|Per student cost of PARE assistance|0.0055|2.0810|165.9700|364.3400|0.0320|0.0000185|
|Score in 4th Grade|0.2518|9.9160|22.6800|11.5000|0.2000||
|Family's Cultural Capital|0.1052|3.9550|53.5900|18.2300|0.1970||
|Teacher's Performance 5th Grade|0.0953|2.3990|51.8600|7.6100|0.1730||
|Teacher's Performance 6th Grade|0.1266|2.9340|42.8300|5.4200|0.1900||
|Director's Quality|0.0928|2.2020|52.2700|7.0500|0.1700||
|Supervision Quality|0.0472|2.7690|63.6900|18.3500|0.1050||
|Parenfts Participation|-0.0626|-2.8090|35.3400|12.2300|-0.0770||
|DUM:MYforRural|-8.8827|-9.0870|0.3000|0.4600|-0.3110||
|DUM:MY for Native|-12.4228|-8.8540|0.2600|0.4400|-0.4350||
|(Constant)|23.0799|4.7220|||||
|Adjusted R Squares||0.2565|||||
|F||67.2897|||||
|N=2114|||||||
|Dependent variable: 6th Grade Spanish test|score||||||
|Estimation method: two-stage least squares|||||||

Source: Own calculations.

18
# 5. Concluding remarks

Exploratory analysis suggests the following ideas. First, students in rural and native schools are way behind others, at least in Spanish, even when school quality and family's cultural capital are taken into account. Second, this disadvantage could be overcome to some extent by providing those schools with PARE-type assistance, focusing on improvement in physical facilities, books and materials, teacher performance incentive, school management and supervision and teacher training. The cost-effectiveness estimates suggest that, despite their imperfection, a 30 percent deficit in test score among rural students can be overcome by roughly doubling the amount of resources per student allocated to those schools to finance the above-mentioned activities. On this point, it is plausible to think that less resources would be needed if school improvement programs were implemented more efficiently and fully.

These conclusions need further verification. It is not clear to what extent these results are applicable beyond the five states under study. Furthermore, due to the limited sample of urban schools, separate analysis of urban children could not be done reasonably well. Finally, fuirther analysis of school effectiveness and parental participation is required.

19
**Variables' Defimitions**

|**NAME**|**DESCRIPTION**|**CONSTRUCTION**|**SCALE**|
|---|---|---|---|
|SCORE IN 6h GRADE|**ESPANOL6:**Scores<br>obtained in the exam of<br>Spanish in**6th**grade.|Scores. The exam has six parts,<br>reading comprehension, use of<br>graphics, writing, language<br>interpretation, literature and<br>writing expression. The grade is<br>given by the percentages of|0-100|
|||correct answers.||
|SCORE IN**4*l**GRADE|**ESPANOL4:**Scores<br>obtained in the exam of<br>Spanish in4thgrade.|Scores. The exam has six parts,<br>reading comprehension, use of<br>graphics, writing, language<br>interpretation, literature and<br>writing expression. The grade is<br>given by the percentages of|0-100|
|||correct answers.||
|DIFFERENCE IN<br>NORMALIZED TEST<br>SCORES BETWEEN**6th**<br>AND 4h" GRADE.|**DIFESP46:**Difference<br>between test scores<br>obtained in exam of<br>Spanish in**6hf**and 4h.|Scores.|0- 100|
|FAMILY EDUCATION<br>BACKGROUND|**CCFAM:**Quantitative<br>indicator of family's<br>cultural capital.|Includes average parents'<br>schooling, lecture habits,<br>television and radio programs<br>and number of books in the<br>house.|0-100|
|FAMILY ECONOMIC<br>BACKGROUND|**NVIIDA:**Family's standard<br>of living index.|Housing quality, purchasing<br>power: transportation services<br>and goods, number of<br>household members.|0-100|
|TEACHER<br>PERFORMANCE (6ff<br>grade)|**DESEMP6:**Quantitative<br>indicator of the teacher<br>performance in 6th grade.|Academic considerations in the<br>improvement of quality of<br>education such as school<br>objectives, teacher's practices in<br>evaluation, attendance, etc.|0-100|
|TEACHER<br>PERFORMANCE**(5th**<br>grade)|DESEMP5: Quantitative<br>indicator of the teacher<br>performance in**6t**grade.|Academic considerations in the<br>improvement of quality of<br>education such as school<br>objectives, teacher's practices in<br>evaluation, attendance, etc.|0-100|
|DIRECTOR'S QUALITY|**DC ACA_1:**Quantitative<br>indicator of director's<br>quality.|Favorable conditions for<br>academic activities, teaching and<br>learning processes. Directors'<br>qualifications and actualization.|0-100|

20
|||Distribution of students in the<br>classrooms.||
|---|---|---|---|
|SUPERVISION<br>QUALITY|**CALI_S_1:**Quantitative<br>indicator of supervision's<br>quality.|Includes annual frequency of<br>visits, duration, occupations of<br>interviewed people and themes<br>discussed.|0-100|
|PARENTS'<br>PARTICIPATION|**APF6:**Quantitative<br>indicator of parents'<br>participation in the school<br>process.|This indicator weighs the<br>attitudes of parents with respect<br>to teachers' attendance, parents'<br>participation in school activities<br>and relevance of parents<br>associations in the school.|0-100|
|UNIT COST|**Unit cost per pupil**|Presents the fixed unit cost per<br>pupil.||
|CHILD'S PART<br>ACADEMIC RECORD|**HIST_ESC:**Index of<br>historical academic record<br>of the student.|Total years in pre-school, total<br>repetition and dropout years.|0-100|
|DUMMY FOR RURAL|**DUMMYR**|Dummy variable: If DUMMYR<br>= 1 then the observation is of<br>rural areas. DUMMYR= 0 for<br>|0 & 1|
|||other cases.||
|DUMMY FOR NATIVE|**DUMMYI**|Dummy variable: If DUMMYR<br>= 1 then the observation is of<br>native areas. DUMMYR= 0 for|0 & I|
|||other cases.||
|FACTOR -||It's a compound index||
|CHARACTERISTICS OF|**FACI_14**|constructed by principal||
|COMMUNITY AND<br>FAMILY||components method. It includes<br>the characteristics of the<br>demand such as family and<br>community background,<br>parental attitude towards and<br>involvement in schooling,<br>academic history, self-esteem,<br>etc.||
|FACTOR-||It's a compound index||
|CHARACTERISTICS OF<br>SCHOOL AND SYSTEM|**FACI_15**|constructed by principal<br>components method. It includes<br>the characteristics of the supply<br>such as teachers, principals and<br>supervisors background,<br>perfornance, attitudes,<br>assiduity, pay, etc., as well as<br>characteristics of the school<br>infrastructure and availability of||

21
|STUDENT' S SELF-<br>SI_MISMO|textbooks, supplies, etc.<br>Student self-esteem index.|
|---|---|
|ESTEEM, 5' GRADE|Student's perception of his own<br>school performance, of his own<br>goals, of other peoples' opinion,<br>and if he thinks that his success<br>depends on himself|

22