metadata: authors: 0: author: Parra Osorio,Juan Carlos
1: author: Wodon,Quentin T.
count: Colombia
docna: 0: docna: Performance of Fe y Alegria high school
            students in Colombia : is it a matter of Fe (faith) or
            Alegria (joy) ?
docty: Policy Research Working Paper
subtopic: Taxation & Subsidies,Private Sector Economics,Education Services Industry,Labor Markets,Educational Sciences
teratopic: Gender,Education
abstracts: cdata!: Fe y Alegria is a catholic network of
            schools that started operations in Colombia in 1971, and in
            2009 served more than 72,000 students in 61 schools. This
            paper assesses the performance of Fe y Alegria secondary
            schools in Colombia using test scores for Spanish and
            mathematics, as well as detailed information on the
            characteristics of the household to which students belong.
            Simple statistics suggest that Fe y Alegria schools perform
            worse than other schools for all years in the sample.
            However, Fe y Alegria schools also cater to poorer students
            who come from disadvantaged backgrounds. Once controls are
            included for student background, Fe y Alegria schools
            actually often perform as well and in some cases better than
            other schools for mathematics and Spanish, thus partially
            reversing the previous finding.
geo_reg_key: 119228,517191
repnme: repnme: Performance of Fe y Alegria high school
            students in Colombia : is it a matter of Fe (faith) or
            Alegria (joy) ?
display_title: Performance of Fe y Alegria high
            school students in Colombia : is it a matter of Fe (faith)
            or Alegria (joy)
listing_relative_url: /research/2010/09/12736336/performance-fe-y-alegria-high-school-students-colombia-matter-fe-faith-or-alegria-joy
topicv3: Public-Private Partnerships m1326334 2206,Private Participation in Education
            m1326520 412,Education m1327048 470,Jobs m1503931 2670,Vouchers m1504820 3553
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Gender,Education
no_of_pages: 22
author_keys: 0: author_key: 782774
1: author_key: 124543
model_extractions: WPS5413 Policy Research Working Paper 5413 Performance of Fe y Alegría High School Students in Colombia Is it a Matter of Fe (Faith) or Alegría (Joy)?

_Juan Carlos Parra Osorio Quentin Wodon1_ The World Bank Human Development Network Development Dialogue on Values and Ethics September 2010
Policy Research Working Paper 5413

# **Abstract**

Fe y Alegría is a catholic network of schools that started operations in Colombia in 1971, and in 2009 served more than 72,000 students in 61 schools. This paper assesses the performance of Fe y Alegría secondary schools in Colombia using test scores for Spanish and mathematics, as well as detailed information on the characteristics of the household to which students belong. Simple statistics suggest that Fe y Alegría schools perform worse than other schools for all years in the sample. However, Fe y Alegría schools also cater to poorer students who come from disadvantaged backgrounds. Once controls are included for student background, Fe y Alegría schools actually often perform as well and in some cases better than other schools for mathematics and Spanish, thus partially reversing the previous finding.

This paper—a product of the Development Dialogue on Values and Ethics, Human Development Network—is part of a larger work program on the role of faith-based organizations in service delivery. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at jparraosorio@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Performance of Fe y Alegría High School Students in Colombia: Is it a Matter of Fe (Faith) or Alegría (Joy)?**

Juan Carlos Parra Osorio and Quentin Wodon<sup>1</sup> Keywords: Education, faith-based schools, public-private partnerships, Fe y Alegría, Colombia JEL categories: I2, L31, H43

## <u>Corresponding author:</u>

Juan Carlos Parra Osorio, World Bank; Email: <u>jparraosorio@worldbank.org</u>

> 1 The authors are with the Development Dialogue on Values and Ethics in the Human Development Network at the World Bank.  The opinions presented in this paper are those of the authors only, and do not necessarily reflect the views of the World Bank, its Executive Directors, or the countries they represent.  The paper was presented at a multi-country training event organized for Fe y Alegría staff in October 2009 in Peru in collaboration with the World Bank Institute and Magis Americas, and at a World Bank education seminar in February 2010 in Washington, DC.  Comments from participants to both events and especially Felipe Barrera-Osorio are gratefully acknowledged.
# **1. Introduction**

An emerging body of evidence suggests that private schools, including faith-based schools, may provide better education services than public schools (e.g., Allcott and Ortega 2009; Altonji et al. 2005; Asadullah et al. 2009; Backiny-Yetna and Wodon, 2009; BarreraOsorio, 2009; Chakrabarti and Peterson, 2009; Cox and Jimenez 1990; Evans and Schwab 1995; González and Arévalo 2005; Hoxby 1994; Hsieh and Urquiola 2006; Wodon and Ying, 2009). In the economic literature, several reasons have been advanced to explain the gains in performance associated with private schools (Epple and Romano 1998; LaRocque and Patrinos 2006; Nechyba 2000; Savas 2000). First, private schools may introduce competition in the education sector and thereby raise overall quality. Second, private providers may have more flexibility than public providers in the management of the schools. Third, to the extent that private providers of education are competitively selected, better providers would emerge in the private as opposed to the public sphere. Fourth, risk-sharing between the government and the private sector may also lead to better provision.

Beyond these economic arguments for a role to be played by faith-based providers, there may also be a number of other potential advantages in having faith-based organizations providing education services. As noted by Belshaw (2005), FBOs have a long-term commitment to their communities and they often reach out to the poorest members of the community.

Through links to sister organizations in other countries, they may benefit from outside funding and expertise. Faith-based schools often emphasize values of respect and consideration for others. In addition, religious leaders often have a moral authority that helps in mobilizing the community’s resources around the schools. Finally, faith-based providers may be more dedicated or altruistic than other providers. As argued by Reinikka and Svensson (2010) in the case of health service provision in Uganda, faith-based providers may not be motivated only or
mostly by profit or perks maximization—they seem to be “working for God.” At the same time however faith-based schools may also suffer from weaknesses, especially if they place the pursuit of their religious mandate ahead of the needs of students in regard to what they need to learn to be successful in today’s world.

In this paper, we assess the performance of Fe y Alegría secondary schools in Colombia using a rich multi-year data set with test scores on a wide range of subjects as well as detailed information on the characteristics of the household to which students belong. Fe y Alegría is a catholic education network founded in 1955, serving almost one million students in 17 countries<sup>2</sup> . It targets excluded population and its work program, according to the organization’s mission, is based on the Christian values of justice, participation, and solidarity. The network started operating in Colombia in 1971, and in 2006 it served more than 72,000 students in 61 schools<sup>3</sup> . Even though academic excellence is not explicitly mentioned in its mission or vision, it is one of the variables Fe y Alegría principals and leaders use for their internal evaluations.

Most observers consider Fe y Alegría to be successful, but few rigorous evaluations have been undertaken until now. This paper was inspired by previous research on the performance of Fe y Alegría in Venezuela conducted by Allcott and Ortega (2009) who use propensity score matching methods to estimate the effects on standardized test scores of graduating from Fe y Alegría in comparison with public schools. They find an average treatment effect on the order of 0.1 standard deviation in mathematics, which is small, but nevertheless statistically significant. They argue that the better performance of Fe y Alegría stems not only from its labor contract flexibility and decentralized administrative structure, but also from the peculiar “family culture” of the schools.

> 2 In chronological order of start of operations: Venezuela, Ecuador, Panama, Peru, Bolivia, El Salvador, Colombia, Nicaragua, Guatemala, Brazil, Dominican Republic, Paraguay, Argentina, Honduras, Chile, Haiti, Spain, and Chad. There are no Fe y Alegría schools in Spain.

> 3 Only formal education is included.
Building on Allcott and Ortega’s (2009) work, the data set and methodology that we use in this paper enables us to test in a more robust way for the performance of Fe y Alegría schools than previous work for three main reasons. First, while Allcott and Ortega (2009) use data on test scores for one year only, we have five years of good data at our disposal, which enables us to test whether differences in performance are robust over time. Second, we have a larger number of subjects on which students are tested. Beyond test scores for mathematics and Spanish, we also look at data on test scores for scientific topics. Third, we have an especially good set of controls regarding the socio-economic characteristics of the families to which the children belong. Finally, we use alternative matching methods (propensity score matching as well as coarse exact matching) to ensure that the results obtained are robust to the choice of method.

Our results confirm previous work suggesting that faith-based schools perform relatively well. Simple statistics on test scores suggest that Fe y Alegría schools perform worse than other schools for all years in the sample. However, Fe y Alegría schools also cater to poorer students. Once controls are included for student background, Fe y Alegría schools actually perform at least as well as, and in several cases better than other schools. Section 2 of the paper describes our data and methodology. Section 3 presents our empirical results. A conclusion follows.

# **2. Data and methodology**

The ICFES (Colombian Institute for the Promotion of Higher Education) test is a standardized test administered by the Ministry of Education that every student has to take during the last year of high school<sup>4</sup> . The test score is used mainly by colleges for admission purposes, and had no implications for high school graduation for the years used in this study<sup>5</sup> . The database provided by ICFES has test scores for every single student taking the test from 1998 to 2006.

> 4 The test can be retaken after graduation from high school.

> 5 It became a requirement for high school graduation in 2009.
Scores for five subjects are analyzed: Spanish, mathematics, biology, chemistry, and physics. Socioeconomic variables at the student level are included in the database from 1998 to 2003<sup>6</sup> . These variables include birth date, gender, level of education of parents, their occupation, number of people in the family, number of siblings, household income, dwelling ownership, and whether the students works. Information on tuition paid is available from 2000 to 2006.

Because our analysis relies on controls and matching techniques to assess the performance of Fe y Alegría schools, we use the data from 1998 to 2003. We restrict the sample using the following criteria: (1) Schools in cities where Fe y Alegría has at least one school in the current year; (2) Students who graduate from high school in the current year or the year before; (3) Students between 15 and 20 years of age; and (4) Students who attended both morning and afternoon sessions of the test.

The data for 2001 seems to be problematic because the number of observations is much smaller than in other years. This then gives us five years of data to work with, for 1998-2000 and 2002-2003. Table 1 contains the total number of observations for Fe y Alegría and other schools for each year in the sample, after imposing the four constraints above. We would have liked to explore different control groups, but there is not enough information in the database to identify, for example, public or private schools, or other school level variables like number of students, number of teachers, and qualification of teachers, among others. That is the reason why our control group includes all non Fe y Alegría schools. The percentage of students in the sample attending Fe y Alegría schools increases from 1.1 in 1998 to 1.5 in 2003; the share of Fe y Alegría schools grows from 1.8 percent in 1998 to 2.5 percent in 2003.

We rely on both traditional propensity score matching and CEM (coarse exact matching) techniques for the estimations (Iacus et al., 2009; King and Stuart, 2007). CEM is a recent

> 6 Socioeconomic information was not collected from 2004 to 2006.
method to form a better control group. It works by defining strata in which all observations have the same values for the observables **_X_** . Consider a sample of n≤N units drawn from a population of _N_ . Let T� denote as an indicator variable for unit _i_ that takes on value 1 if unit _i_ is a member of the treated group (Fe y Alegría in our case) and 0 if _i_ is a member of the control group (other schools). The observed outcome (test scores in our case) variable is Y� = T�Y�(1) + (1 −T�)Y�(0), where Y�(0) is the potential outcome for observation _i_ if the unit does not receive treatment and Y�(1) is the potential outcome if the (same) unit receives treatment. For each observed unit, Y�(0) is unobserved if _i_ receives treatment and Y�(1) is unobserved if _i_ does not receive treatment; this is known as the “missing observation problem” in the evaluation literature.

To compensate for the fact that treated and control groups are not necessarily identical before treatment, matching estimators aim to control for pre-treatment covariates. Denote X= (X�, X�, ⋯, X�) as a _k_ -dimensional data set, where each X� is a column vector of observed values of pre-treatment variable _j_ for the _n_ sample observations. That is, X= �X��, i= 1, ... , n, j= 1, ... , k]. Let T= {i: T� = 1} be the set of indexes for the treated units and n� = #T be a count of the elements of this set; similarly C= {i: T� = 0}, n� = #C for the control units, with n� + n� = n. Denote by m� and m� the number of treated and control units matched by CEM. Let MT ⊆T and MC ⊆C be the sets of indexes of the matched units in the two groups.

After coarsening, the CEM algorithm creates a set of strata (cells), say s∈S, each with the same values of _X_ . Observations in strata that contain at least one treated and one control unit are retained; all other observations are dropped from the sample. We denote by T<sup>�</sup> the treated units in stratum _s_ , and by m�� = #T� the number of treated units in the stratum, similarly for the control units, i.e. C<sup>�</sup> and m�� = #C �. The number of matched units is, respectively for treated and
� � controls, m� = ⋃�∈�m� and m� = ⋃�∈�m�. To each matched unit _i_ in stratum _s_ , CEM assigns the following weights: Unmatched units receive weight w� = 0. After the final sample is constructed using CEM, we just compare, within each stratum, the test scores for treated observations with the average test score for the control observations.

# **3. Results**

Tables 2 and 3 provide basic statistics on the characteristics of students in both Fe y Alegría and other schools. On average, students in Fe y Alegría schools live in households with a lower income per capita, whose parents have lower education levels, have higher propensity to own their dwelling that is inhabited by slightly bigger households. All variables used in the estimations have been transformed into categorical variables.

Several studies have found negative correlations between students’ performance and lower per capita income, parents’ education level, and higher household size. This is likely then to lead to lower average test scores for students in Fe y Alegría schools, and it is indeed what is observed for every subject and every year. Table 4 presents the average scores in Fe y Alegría schools and other schools. The results indicate that students in other schools do better than students in Fe y Alegría schools in every subject and for every year in the sample. The gap in Spanish and mathematics is at 1.0 to 4.5 percentage points, and the gaps for biology, chemistry, and physics are between 2.0 and 7.0 percentage points (see Figure 1).
In order to control for student characteristics, we rely on matching techniques<sup>7</sup> . If test scores are independent of the decision of whether to attend a Fe y Alegría school given the observables, comparing scores between Fe y Alegría students and other schools using matching techniques will give an unbiased estimate of the difference in test scores attributable to the type of school attended. However, including all observations in the sample might make the results highly model-dependent if the control and the treatment groups are not similar enough. The ideal situation is to have both groups with identical distributions for all observable (and unobservable) characteristics. In order to improve the balance of our sample, that is to have a control group that is more similar to the treated group, we select the final sample used for our estimations using coarse exact matching or CEM (see Section 2; Iacus et al., 2009; King and Stuart, 2007). In order to run the CEM model, we include as control variables the parents’ education level, the parents’ occupation, household size, per capita income, whether the student works, dwelling ownership, student gender, student age, and location (city) dummies<sup>8</sup> . The CEM algorithm defines strata, each with the same value of the observables. Only strata with at least one observation from the treated group and one from the control group are used. All other observations are discarded<sup>9</sup> . As a test of robustness, we also use propensity score matching, with a probit regression model whose estimates are provided in annex.

Key results are presented in table 5 for all subjects, as well as for mathematics and Spanish in Figures 2 and 3. Figure 2 provides the percentage difference in test scores based on CEM. The hollow bars represent differences that are not statistically different from zero. In Figure 3, the same is done with the results from propensity score matching. We tend to favor the

> 7 One way to control for observable variables like income, parents’ education, household size, gender, among others, would be to include them in a simple regression model, such as OLS, with the regression including a dummy for Fe y Alegría as one of the explanatory variables. This however would assume that treatment effects are similar for the sample as a whole, which may not be appropriate.  Using matching techniques relaxes that assumption. 8 We use level dummies for all variables to capture potential nonlinearities in the model.

> 9 On average, 40 percent of the observations in the control group (other schools) are dropped after CEM.
results based on CEM, since the matching is more precise in that case, but the results are fairly similar using both methods. Consider first mathematics and Spanish, which are more commonly used for measuring performance (both in the literature and in Colombia) than scientific subjects. It appears that except in 2003, Fe y Alegría schools do as well or better than other schools. However, for physics, chemistry and biology, the performance of Fe y Alegría schools tends to be lower (the only exception is for physics in 1998 using the CEM method).

Looking at the Figures and estimates, one may wonder whether there is a trend in the results for mathematics and Spanish, with better performance for Fe y Alegría schools in earlier years in comparison to other schools, no statistical difference in performance in the middle years, and lower performance in the last year of data. Because the sample of schools differs from year to year, with a larger number of Fe y Alegría schools over time, this could potentially be explained by the fact that perhaps new Fe y Alegría schools could perform less well than older schools, thereby pulling overall results for the network down over time. The data however does not confirm this hypothesis. We redid the estimations by restricting the school sample to remain the same in 2003 as per the 2000 sample (given that the number of schools in 1998 and 1999 is smaller), and this did not affect the results for 2003 in that they remained essentially unchanged, as well as statistically significant in terms of the difference between Fe y Alegría and other schools for that year. Thus we cannot explain the apparently lower performance for Fe y Alegría in 2003 by the change in the sample of schools covered over time, but given that we have only one year of data where Fe y Alegría schools perform at a lower level than other schools, we think that it would be premature to assert that there is indeed solid evidence for a trend towards a lower level of performance among Fe y Alegría schools over time.
# **4. Conclusion**

This paper quantified the differences in high school test scores between students in Fe y Alegría schools and other schools in Colombia between 1998 and 2003. We find that since Fe y Alegría caters, on average, to poorer students and students with lower-educated parents, simply comparing average test scores might lead to wrong conclusions regarding the performance of the schools. We used a recent method (coarse exact matching) to form a better control group (balancing) and found that for mathematics and Spanish, most negative gaps for Fe y Alegría students that were found across the years and subjects when comparing averages, either vanish or become gains after balancing in 4 out of 5 years. For robustness purposes, we also estimated performance using propensity score matching, and the results were similar. On the other hand, Fe y Alegría students tend to do less well in physics, chemistry and biology, although the difference after matching with other schools is significantly smaller than the difference before matching.