metadata: authors: 0: author: Jha,Shreyasi
1: author: Khattri,Nidhi
2: author: Ling,Cristina Marosan
count: Philippines
docna: 0: docna: The effects of school-based management in
            the Philippines : an initial assessment using administrative data
docty: Policy Research Working Paper
subtopic: Educational Sciences,Educational Institutions & Facilities,Educational Populations,Education for Development (superceded),Educational Policy and Planning - Decentralization,Educational Policy and Planning - Textbook,Effective Schools and Teachers,Education For All
teratopic: Education
abstracts: cdata!: This paper estimates the effect of
            school-based management on student performance in the
            Philippines using the administrative dataset of all public
            schools in 23 school districts over a 3-year period,
            2003-2005. The authors test whether schools that received
            early school-based management interventions (training in
            school-based management and direct funding for school-based
            reforms) attained higher average test scores than those that
            did not receive such inputs. The analysis uses school-level
            overall composite test scores (comprising all subject areas
            tested) and test scores in three separate subject areas:
            English, math, and science. Their preferred estimator,
            difference-in-difference with propensity score matching,
            shows that the average treatment effect of participation in
            school-based management was higher by 1.5 percentage points
            for overall composite scores, 1.2 percentage points for math
            scores, 1.4 percentage points for English scores, and 1.8
            percentage points for science scores. These results suggest
            that the introduction of school-based management had a
            statistically significant, albeit small, overall positive
            effect on average school-level test scores in 23 school
            districts in the Philippines. The paper provides a first
            glimpse of the potential for school-based management in an
            East Asian context based on available administrative data.
            The authors suggest that the next order of research is to
            answer policy-related questions regarding the reforms: what
            aspects of the reform lead to desired results; are there
            differential effects across subpopulations; and what are the
            potential downsides to the reforms? The Philippines is
            embarking on a nation-wide implementation of school-based
            management and the authors recommend that mechanisms for
            rigorous evaluations be advanced simultaneously. Such
            evaluations should not only provide more accurate estimates
            of the effectiveness of the reforms, but also help answer
            policy-related questions regarding design and implementation
            of those reforms in different socio-cultural contexts.
geo_reg_key: 517198,517191
repnme: repnme: The effects of school-based management in
            the Philippines : an initial assessment using administrative data
display_title: The effects of school-based
            management in the Philippines : an initial assessment using
            administrative data
listing_relative_url: /research/2010/03/11992660/effects-school-based-management-philippines-initial-assessment-using-administrative-data
topicv3: Teachers Management m1326119 1523,Textbooks m1326802 2129,Education Districts m1327034 2141,Education m1327048 470,School-Based Management m1327730 2143
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Education
no_of_pages: 29
author_keys: 0: author_key: 642133
1: author_key: 560271
2: author_key: 657368
model_extractions: WPS5248 Policy Research Working Paper 5248 The Effects of School-based Management in the Philippines An Initial Assessment Using Administrative Data _Nidhi Khattri Cristina Ling Shreyasi Jha_ The World Bank Independent Evaluation Group East Asia Education Sector Unit & World Bank Institute March 2010
Policy Research Working Paper 5248

# **Abstract**

This paper estimates the effect of school-based management on student performance in the Philippines using the administrative dataset of all public schools in 23 school districts over a 3-year period, 2003–2005. The authors test whether schools that received early schoolbased management interventions (training in schoolbased management and direct funding for school-based reforms) attained higher average test scores than those that did not receive such inputs. The analysis uses schoollevel overall composite test scores (comprising all subject areas tested) and test scores in three separate subject areas: English, math, and science. Their preferred estimator, difference-in-difference with propensity score matching, shows that the average treatment effect of participation in school-based management was higher by 1.5 percentage points for overall composite scores, 1.2 percentage points for math scores, 1.4 percentage points for English scores, and 1.8 percentage points for science scores. These results suggest that the introduction of schoolbased management had a statistically significant, albeit small, overall positive effect on average school-level test scores in 23 school districts in the Philippines. The paper provides a first glimpse of the potential for school-based management in an East Asian context based on available administrative data. The authors suggest that the next order of research is to answer policy-related questions regarding the reforms: what aspects of the reform lead to desired results; are there differential effects across subpopulations; and what are the potential downsides to the reforms? The Philippines is embarking on a nationwide implementation of school-based management and the authors recommend that mechanisms for rigorous evaluations be advanced simultaneously. Such evaluations should not only provide more accurate estimates of the effectiveness of the reforms, but also help answer policyrelated questions regarding design and implementation of those reforms in different socio-cultural contexts.

This paper—a joint product of East Asia Education Sector Unit, Independent Evaluation Group; and the World Bank Institute—of a broader Bank-wide effort to promote rigorous and analytical approaches to monitoring and evaluation. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at nkhattri@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **The Effects of School-based Management in the Philippines: An Initial Assessment Using Administrative Data**

Nidhi Khattri<sup>1</sup> Cristina Ling Shreyasi Jha

**Keywords:** school-based management, decentralization, evaluation, propensity score matching, administrative data

> 1 Nidhi Khattri, Senior Evaluation Officer, Independent Evaluation Group, The World Bank, Washington D.C. email: nkhattri@worldbank.org; Cristina Ling, Evaluation Officer, World Bank Institute Evaluation Group, The World Bank, Washington D.C. email: <u>cling@worldbank.org; Shreyasi Jha, Adjunct Professor, The Wilf Family</u> Department of Politics, New York University, New York. Email: <u>sj46@nyu.edu.</u> We would like to thank the Department of Education of the Philippines and Professor H. Nuqui, University of the Philippines, for providing the data and information on SBM.  The authors also gratefully acknowledge the support of Dingyong Hou (Senior Operations Officer, EAPHD, and  Task Team Leader for the Third Elementary Education Project), Lynette Dela Cruz Perez (Senior Operations Officer, EASHE), Christopher Thomas (Manager, EAPHD), and Eduardo Velez Bustillo (EASHE).  This work was partially supported by the Education Program Development Fund.  We are indebted our reviewers for providing helpful comments and suggestions on previous drafts.  All remaining errors are ours alone.  The opinions and conclusions expressed in this paper do not necessarily reflect those of the Executive Directors and officials of the World Bank or of its member governments.
# **1.  Introduction**

Decentralization is a key feature of institutional reform throughout the world.<sup>**2**</sup> The main argument underpinning decentralization policies is that they empower people to be part of the local decision-making process –they improve government performance by alleviating information asymmetries and costs by bringing decision-making closer to the people concerned. However, decentralization can also worsen the provision of public goods in the presence of externalities, lack of technical capabilities by local governments, or capture of lower-level administration by local elites.<sup>3</sup> In the context of the education sector, decentralization typically includes one or more of the following features: decentralized revenue generation, curriculum design, school administration, and teacher hiring and management. Decisionmaking authority for these types of functions is devolved to regional/municipal governments or to schools themselves. The policy of allowing schools autonomy in decisions in these areas is referred to as schoolbased management (SBM), school based governance, or school self management<sup>4</sup> ,<sup>5</sup> . Responsibility and decision-making over different types of school operations are transferred to individuals at the school level, who in turn must conform within a set of centrally or state-level determined policies.

The popularity of SBM is evidenced by the large number of development agencies promoting it as a key component of the decentralization reforms and the growing number of countries that have adopted aspects of this approach. SBM reforms began in the 1970s in Australia. Since then, a wide range of countries have experimented with or introduced SBM in all regions of the world, including Hong Kong (China), Indonesia, El Salvador, Nicaragua, Kenya, Kyrgyz Republic, Nepal, Paraguay and Mexico.<sup>6</sup> Nevertheless, the impact of SBM on education quality, including student outcomes, remains a contentious issue, with some researchers arguing that SBM leads to enhanced educational outcomes,<sup>7</sup> while others contending that SBM leads to the deterioration of educational quality especially among the weakest schools.<sup>8</sup> The range of SBM approaches and the contexts in which they are implemented makes the debate about SBM quality an intricate one. The evaluation of SBM is complicated by the diversity of approaches to and elements of decentralization that collectively constitute “SBM” and by the institutional and socio-cultural contexts in which they are implemented. Nonetheless, some studies in recent years have found that SBM reforms are associated with improved education outcomes and processes (e.g.,

> 2 King and Ozler, 1998.

> 3 Galiani and Schargrodsky, 2001

> 4 De Grauwe, 2004

> 5 Gertler, Patrinos, and Codina, 2006

> 6 See “What is SBM?” World Bank (2007)

> 7 e.g., Gertler, Patrinos, and Codina, 2006

> 8 Bardhan, 2002.

2
Skoufias & Shapiro, 2006; Sawada & Ragatz, 2005; Gunnarsson et al., 2004; Eskeland & Filmer, 2002). However, rigorous evidence base for the effectiveness of SBM in boosting student performance is thin. A recent review of the empirical literature on SBM since 1995 indicates that only 14 studies utilized rigorous methods to assess the impact of SBM, and only six reported positive impacts on students’ test scores (Barrera-Osorio et al., 2009) . Eleven studies are country-specific from Latin America, one from Kenya, and two exploit data from multiple countries. No empirical evidence is available from East Asia.

This paper contributes to the small but growing empirical literature on the effectiveness of SBM by extending the research to East Asia. The paper provides an initial analysis of the potential of schoolbased management for improving educational outcomes in the Philippines, using aggregated school-level test scores and administrative data. The data are quite limited and do not allow for a thorough analysis of the processes and approaches through which SBM reforms affect outcomes. Nonetheless, the paper attempts to use existing data to answer an initial question: did the introduction of SBM lead, on average, to enhanced educational outcomes? It also demonstrates how administrative data can be mined for exploratory assessments of potentially larger programs.

The remainder of the paper is organized as follows: the next section provides a brief overview of the SBM program in the Philippines; section 3 outlines the methodological approaches used for the analysis; section 4 describes the data; section 5 discusses the empirical results; and section 6 provides a summary of our conclusions, discusses the limitations of the study, and highlights the implications for future, more rigorous SBM evaluations in the Philippines.

# **2.  SBM Program in the Philippines**

SBM was implemented in between 2003 and 2005 in 23 districts participating in the Third Elementary Education Project (TEEP) supported by the World Bank.<sup>9</sup> The project provided funding for school infrastructure, training, curriculum development, and textbooks. SBM was introduced as an integrating framework for obtaining school-level project inputs and building school capacity for education planning and program implementation beginning in school year 2003-04. Schools participating in SBM were required to design a five-year School Improvement Plan (SIP) in partnership with parents and the community using data such as student achievement and students’ learning needs assessments, with the school principal or head teacher leading the process. Based on the SIP, schools developed an Annual

> 9 Although some other school districts in the Philippines had ongoing SBM initiatives, they are not included in this analysis as insufficient information is available about the nature and timing of those initiatives.

3
Implementation Plan (AIP) at the beginning of the school year and a report card to be shared with the community at the end of the school year. Project inputs for infrastructure, training, textbooks, and so forth, were partially based on the SIP. Principals and head teachers received training in leading the development and implementation of the SIP and the AIPs in collaboration with teachers and key members of the larger community. SBM schools also received funds for maintenance and operating expenses directly in cash rather than in kind, as had been the case previously. These cash funds could be used by the schools based on their AIP, The cash allocation was based on a formula that provided each school with a flat amount of funds plus a prorated figure based on the number of student and teachers as well as other criteria, such as percentage of indigenous student population in the school. Schools not participating in the SBM received no SBM-related training and no cash funds, and they were not required to develop SIPs and AIPs.

The SBM was designed to improve student outcomes through two main venues: by empowering the school community to identify education priorities and to allocate the school maintenance and operating budgets to those priorities (such as curriculum enrichment programs); and by enhancing transparency and accountability through the annual implementation plans and school report cards. However, the SBM program articulated no explicit assumptions regarding the timeframe within which improvements in student achievement were expected to take place. Systematic data on the level of uptake and implementation of the key features of the reforms are also not available.

The SBM training, funds, and requirements were rolled out in three batches and eventually covered almost all (84 percent) of the 8,613 schools in the 23 project districts. The first batch comprised 1,666 schools in 2003-2004, largely because they were perceived to be more capable, although no explicit assignment mechanism was designed.<sup>10</sup> The next batch of 2,700 schools was targeted for SBM rollout in 2004-2005, and another batch of 1,529 was included in 2005-2006.

# **3.  Analytic Approach**

We assess the effects of SBM on student performance using average school-level test scores from all schools and school-level indicators between 2003 and 2005 from the 23 TEEP districts in the Philippines. We base our analysis on a pipeline comparison strategy. Since we have student achievement data over a three year period, but schools were inducted into the program in three batches over time, we identify the

> 10 No written materials were available on the identification strategy.  Project officials interviewed as part of this paper mentioned that schools were chosen on the basis of the perceived strength of their capabilities.

4
treatment group as the _first_ batch of schools that received SBM funds in 2003-2004. The treatment group consists of the schools that did not receive SBM funds and training in 2003-2004. Therefore, for this study, we operationalized SBM batch one schools as the _treatment group_ (n=1666) and all other schools (batches that received the SBM intervention later) as the _control group_ (n=3501).<sup>11</sup> In our analysis, the treatment group (batch one) therefore had exposure to SBM for a period of two years, 2003-04 and 200405.

Because selection into SBM was not voluntary, there is likely to be placement bias in our control and treatment groups. We therefore use two non-experimental evaluation techniques to estimate program impact: difference-in-difference estimators and difference-in-difference with propensity score matching estimators. Using nonparametric kernel matching techniques, we correct for potential sources of bias.

Specifically, we tested whether the composite test scores and test scores individually in three subject areas (math, science and English) were higher after SBM was introduced for schools in the treatment group compared to those in the control group.

# **4.  Data and Statistical Specification**

We use school administrative data from all 5,167 schools in the 23 TEEP districts to examine the effect of SBM program on student test scores. The dataset includes information on school personnel, students, facilities, and enrollment and dropouts for all schools in the district over a 3-year period, 2002-2003 to 2004-2005. The data were collected by the Philippines Department of Education as part of the management of the Third Elementary Education Project (TEEP) of which SBM was one component.

Student achievement is measured using National Achievement Tests conducted in 2002-03 (year one – pre-intervention) in Grade 4, 2003-04 (year two) in Grade 5, and 2004-05 (year three, post-intervention) in Grade 6 in English, mathematics, science. However, only the school-level Mean Percentile Score (MPS) data are available in the dataset, limiting the analysis to the school level. We utilize the composite MPS computation (based on tests in all subjects), as well as the English, mathematics, and science MPS for 2002-03 (pre-intervention) and 2004-05 (post-intervention).

> 11 The number of schools in the analysis is not the same as the number of schools included in the SBM program due to missing information.  The difference is 728 schools.

5
Our method of analysis consists of two quasi-experimental evaluation techniques. We have data on test scores over three years for all schools. We also have time-invariant data on school characteristics. Once we determined the control and treatment groups as described in the preceding section, the preferred estimator to use in this context was the _difference-in-differences (DID)_ estimator. However, in order for the DID estimator to provide credible estimates of the program impact, it should be free of any biases inherited by pre-existing (pre-intervention) differences between treatment and control groups. Because of problems incurred with using the DID estimator due to way in which SBM program was rolled out (i.e., no clear assignment mechanism), we also estimate and discuss DID in conjunction with _propensity score matching (PSM)_ to estimate average treatment effects.

# **_(i) Difference-in-differences (DID) estimation_**

The ideal experiment for identifying the effects of SBM on student test scores would have been to randomly assign schools to control and treatment groups. The SBM effect could then have been estimated by comparing schools that received SBM with their peers in the comparison group. Unfortunately, such a design is not feasible in this case because of the nature of the SBM program rollout. Therefore, we assigned the set of schools that received early intervention in 2003 as the treatment group. The control group comprised all other schools that did not receive SBM support in 2003 – i.e., schools that received the intervention later. The SBM effect on student test scores can be evaluated as follows: SBM effect = ( _TestScores SBM_ . _POST_  _Testscores SBM_ , _PRE_ )  ( _Testscores NONSBM_ , _POST_  _TestScores NONSBM_ . _PRE_ ) --------------------(eq. 1) Converting equation (1) above to an estimating model, we get: _Ys_ , _t_   0   1 _Yeart_   2 _SBM s_   3 ( _Yeart_ * _SBM s_ )   4 _Z ts_   _p_   _ts_ -------------------(eq. 2) where Y is the outcome measure. We include school-level composite test scores as well as test scores in math, science, and English as outcome measures. Year is a dichotomous variable that equals 1 for postintervention and equals 0 for pre-intervention. SBM is a dichotomous variable that is equal to 1 indicating that a school received SBM support during all of the treatment years ( _t_ =2002-2004). Z is a vector of school characteristics.  _p_ is division-specific fixed effects intended to capture division-specific aggregate fixed effects correlated with schooling outcomes (demographic trends or changes in government, for 6
> example).  _t_ , _s_ is the school level error term that includes all the unobserved school characteristics  that we assumed are uncorrelated with the explanatory variables for the time being.<sup>12</sup>

The vector of school characteristics includes total enrollment, total dropout, student-teacher ratio, textbooks per student, teacher manual per teacher, school type, school head type, whether the school was an elementary leader school (a school considered to be a leader among others in the district) and whether the school had received SIIF funds.<sup>13</sup> School type consists of four categories: complete combination and multi-grade schools; complete large and medium school; complete small schools; and primary schools. School head type consisted of three categories: head principal, head teacher, and teacher in charge. Table 1 summarizes the descriptive statistics for all schools and by treatment status of the school.

 0 is the intercept showing the average test scores of the SBM and non-SBM schools;  1 shows the change between 2002-03 and 2004-05;  2 is the difference in the treatment and control group; and  3 is the DID estimator capturing the differences between the control and treatment groups, before and after the intervention. More specifically, it measures the SBM effect as the change in average school-level test scores among two groups before and after the 2003 SBM intervention. These groups included: the “first batch,” or the schools that received SBM first (treatment group) to schools that received the intervention later and those that did not receive the intervention at all (comparison group).

In order to obtain the DID estimates, we created a panel dataset from our original dataset of 5,167 schools. Since we had test scores for all three years, we simply replicated this data for every year to obtain a panel dataset with 15,501 observations. In this dataset, each school is observed in each of the three years (2002-03 to 2004-05).

In order to use the DID estimates in this context, we need to be certain that the difference in postintervention outcomes between SBM and non-SBM schools would have been identical in the absence of the intervention. However, this assumption is impossible to test because we do not actually observe the counterfactual. We can nonetheless test whether pre-intervention and “mid-term” educational outcomes

> 12 The intervention might alter the number of children enrolling in a particular school. If as a consequence the distribution of students’ skills changes in treatment schools (with respect to control schools), then the program impact estimates are likely to be biased. We will explore the existence of this bias in section five when discuss the caveats. The characteristics of the average student in the school are also included in the error term because of lack of data on individual students’ characteristics. 13 School Improvement and Innovation Facility that granted funds to schools based on proposals for school improvement programs.

7
under study were similar between the treatment group and the proposed control group. If the preintervention and “mid-term” (past baseline and probably without implemented intervention) outcome measures were not significantly different between treatment and control schools, there is no compelling reason to believe they would be significantly different in the post-intervention periods had the SBM program not been put in place. However, if we observe changes between pre-intervention and mid-term, we can consider the possibility that the two groups would have diverged with respect to their outcomes even in the absence of the SBM intervention. We can test this assumption for the proposed treatment and control groups by running the following equation on the pre-SBM data (i.e. 2002-2003):

-------------------(eq. 3) Y is the outcome measure including:  school level composite test scores and test scores individually in math, science, and English. _Year_ is a dummy variable that takes value = 1, if year =2003-04 and =0 if year=2002-03 (both pre-intervention and implementation years). SBM is a dichotomous variable that is equal to 1 if the school _s_ received SBM support ( _SBM_ =1). As in eq. (2),  _p_ is intended to capture

> division-specific aggregate fixed effects and  _t_ , _s_ is the error term. We are interested in two issues: first, was there a pre-existing difference between SBM and non-SBM schools before the intervention; and two, did the difference increase over time (between 2002-03 and 2003-04). The coefficient for the interaction term can be interpreted as follows:  if the coefficient  2 =0, then the pre-SBM outcomes for schools that would eventually receive SBM funds are not significantly different from those in the control group (i.e., schools that did not receive SBM funds in 2003).

We run equation (3) separately for years 2002-03 and 2003-04. When year=2002-03 (pre-intervention), we do not have a coefficient for  1 .<sup>Table 2 reports the results from equation (3). For 2002, we report</sup> coefficient  2 which shows the difference between SBM and non-SBM schools at the baseline (2002-03). For 2003-04 (mid-term), we report  2 which shows the increase in test scores of SBM schools compared to non-SBM schools in 2003-04 compared to the baseline (2002-03). For both years Model A is a simple OLS and<sup>Model B includes division fixed effects. For overall test scores, Model B in Table 2 shows that</sup>  2 =2.0, implying that overall test scores for schools that would eventually receive SBM funds in 2003 were about 2 percentage points higher than non-SBM schools in 2002-03. In 2003-04 ( mid-term), the 8
difference in overall test scores between SBM and non-SBM schools was about 3.0, implying that overall test scores for SBM schools 3 percentage points higher than non-SBM schools in 2003 compared to 2002.

Table 2 shows there are significant differences in the pre-intervention period in test scores (math, science, English and overall) across schools that would eventually receive SBM and those that did not receive SBM in 2003. Based on these results, it appears that program placement bias is a serious concern in this context, and that difference-in-difference estimates may not give us unbiased estimates of the program effect. Therefore, in addition to difference-in-difference estimates, we also obtain propensity score matching estimates as a means of achieving unbiased identification of the SBM effect.

# **_(ii) Propensity score matching (PSM) estimation_**

One of the potentially most serious problems in any evaluation study is the occurrence of placement bias. In this case, the problem arises because we would like to know students’ test scores with and without the SBM intervention. The results in Table 2 indicate that placement bias is a serious concern in this case because SBM and non-SBM schools are substantially different even in the absence of the treatment. The matching approach is one possible solution to the selection problem. This method originated from the statistical literature and shows a close link to the experimental context<sup>14</sup> . The basic idea is to identify within a large group of non-participants those individuals who are similar to the participants in all relevant pre-treatment characteristics X. The assumption is that once this is done, differences in outcomes between the matched control group and the treatment participants can be attributed to the program.

Since conditioning on all relevant covariates is limited in case of a high dimensional vector _X_ (`curse of dimensionality'), Rosenbaum and Rubin (1983) suggest the use of so-called balancing scores _b_ ( _X_ ), i.e., functions of the relevant observed covariates _X_ such that the conditional distribution of _X_ given _b_ ( _X_ ) is independent of assignment into treatment. One possible balancing score is the propensity score, i.e., the probability of participating in a program given observed characteristics _X_ . Matching procedures based on this balancing score are known as propensity score matching (PSM) and we will use PSM matching in conjunction with DID as the preferred estimator in this paper. PSM proposes that all pre-treatment characteristics of a unit of observation is summarized into a single index (the propensity scores) and the units of observations are then matched on their propensity scores.

According to Skoufias and Shapiro (2006), the propensity score is defined as the conditional probability of receiving a treatment (receiving SBM) given pre-treatment characteristics _X, i.e._

> 14 Propensity Score Matching (PSM) was proposed by Rosenbaum and Rubin (1983).

9
# _p(X)_ ≡ Pr ( _D_ = 1| _X_ ) = _E_ ( _D_ | _X_ )

------------------------------(eq. 4)

where _D_ is the indicator for receiving treatment (=1 if SBM, =0 if non-SBM). As shown by Rosenbaum and Rubin (1983), if the propensity score _p(X)_ is known, then the PSM estimator for ATT (`average treatment effect on the treated’) can be estimated as follows:

------------------------------(eq.5) The potential outcomes are defined as _Yi_ ( _Di_ ) for each individual _i_ , where _i_ = 1 _....N_ and _N_ denotes the total population. To put eq. 5 in words, the PSM estimator is simply the mean difference in outcomes over the common support, appropriately weighted by the propensity score distribution of participants.

When estimating the propensity scores, two choices must be made. The first one concerns the model to be used for the estimation, and the second one requires identifying the variables to be included in the model. We use a logit model to estimate propensity scores. The variables in the model were selected to satisfy the _conditional independence assumption (CIA)._ The matching strategy builds on the CIA, requiring that the outcome variable(s) must be independent of treatment, conditional on the propensity score. Hence, implementing matching requires choosing a set of variables X that credibly satisfy this condition. Only variables that influence simultaneously the participation decision and the outcome variable should be included. We included five explanatory variables in the logit model: student-teacher ratio; size of school (total enrollment in 2002 base year); school type (incomplete vs. complete schools & combination/multigrade schools vs. mono schools); type of school head (principal vs. teachers and teacher-in-charge).

The results from the logit model are presented in Table 3. We impose the common support area condition to ensure that any combination of characteristics observed in the treatment schools can also be observed among the control group.<sup>15</sup> We then apply the nonparametric kernel matching technique to match schools based on their propensity scores within school divisions. We matched schools at the division level because the SBM intervention took place within school divisions. Kernel matching uses weighted averages of all individual cases in the control group to construct the counterfactual outcome. Thus, one major advantage of these approaches is the lower variance which is achieved because more information is used.

> 15 Given that common support area condition is met, PSM approach eliminates a substantial amount of selection bias that would alter conclusions in interpretation (Heckman, Hidehiko, Smith and Todd, 1996).

10
Since we do not condition on all covariates but on the propensity score, it has to be checked if the matching procedure is able to balance the distribution of the relevant variables in both the control and treatment group. Several procedures have been suggested to check the matching quality. The basic idea behind these approaches is to compare the situation before and after matching, and verify whether any differences remain after conditioning on the propensity score. If differences exist, matching on the score was not (completely) successful and remedial measures must be taken. We use a stratification test proposed by Dehejia and Wahba (2002) that divides observations into strata based on the estimated propensity score, such that no statistically significant difference remains between the estimated propensity score means of the treatment and control groups. We then use t-tests within each stratum to test if the distribution of _X_ -variables is the same between both groups. Table 4 presents the absolute t-scores for the X-variables across SBM and non-SBM school by quintiles. We notice that across variables there are no significant differences between SBM and non-SBM schools on most variables except total enrollment.<sup>16,17</sup> Finally, given the availability of data over three years we use the differences-in-difference estimator (developed by Heckman et al. in 1997, 1998a,b) to calculate the average treatment effect _(ATE)._ This estimator compares the before-after test scores of SBM schools with the corresponding before and after changes among non-SBM schools, conditional on covariates X. This builds on the simple differences-indifference statistic by controlling for covariates X and estimating the differences using nonparametric methods. The advantages of this estimator are similar to the advantages of the DID regression – it eliminates any unobserved factors that vary between observations, although not over time. This matching estimator allows selection on unobservables as long as the unobservable factors do not change between observations over time. Given the aforementioned advantages of the simple regression-based DID estimator, the DID with matching is our preferred estimator of program impact.

# **5. Results**

Table 5 presents the simple means of school level test scores across SBM and non-SBM schools, before and after the intervention in 2003, without controlling for other explanatory variables. These figures do not control for any selection bias. Given that there is a strong possibility of selection bias, these estimates are helpful in showing trends and observing overall differences. However, we cannot make any claims of

> 16 Some of the cells do not show t-scores because there were no observations for that particular variable in that cell.

> 17 We tried several combinations of variables, but due to a limited dataset and number of available variables, kernel matching using the variables noted above provided the closest matches.

11
causality based on these comparisons. We examined test scores along four categories: math, English, science and overall (composite of all test scores).

Comparing SBM schools over time, we find that SBM schools scored higher in 2004 (year 3) in all areas including: math by 16.57 percentage points, science by 11.74 percentage points, English scores by 18.57 percentage points and in overall scores by 16.16 percentage points. We find similar increases in test scores for non-SBM schools, although the increases are not as large as they are for SBM schools. Math scores increased by 15.46 percentage points, science scores increased by 10.37 percentage points, English scores increased by 16.66 percentage points and overall scores increased by 14.65 percentage points.

Cross-sectionally, we find that SBM and non-SBM schools performed relatively similarly in 2002 (preintervention) (see Table 5). In 2005, the SBM schools performed better than did non-SBM schools. For example, math scores between SBM and non-SBM schools increased to 1.58 percentage points, science scores were 2.03 percentage points higher, English scores were 2.20 percentage points higher and overall scores are 1.98 percentage points higher for SBM schools compared to non-SBM schools.

A simple difference-in-difference comparison (without controlling for any factors) between SBM and non-SBM schools shows that test scores increased more rapidly among SBM schools. For instance, math scores increased by 1.12 percentage points for SBM schools, science scores increased by 1.36 percentage points, English scores by 1.91 percentage points and overall scores by 1.51 percentage points. Although these results point toward larger increase in test scores, we cannot make any causal inferences based on these estimates.

Table 6 presents the estimates of the average treatment effect derived from three multivariate analysis techniques. Since we are primarily interested in testing whether the introduction of SBM had an effect on test scores, we present only the coefficients associated with the variable measuring program impact. The detailed regression results are presented in the appendices.

The first column presents the estimates of the average treatment effects obtained from running an ordinary least squares (OLS) model of equation (2) without any division level fixed effects. We find that there was a significant increase in overall test score (1.15 percentage points), as well as an increase in test scores in specific subject areas: science (1.37 percentage points) and English (1.14 percentage points); the increase in math was not significant.

12
Next, we ran equation (2) including division level fixed effects and find more or less similar results. We find that there was a statistically significant increase in overall combined test scores (1.06 percentage points) as well as an increase in science (1.28 percentage points) and English scores (1.05 percentage points). As with the OLS model, the increase in math scores for SBM schools compared to non-SBM schools was not significant.

Finally, we present the results from DID estimators based on kernel matching. Generally, we find that the size of the impact was higher relative to DID without matching. Using kernel propensity score matching, we find that participating in the SBM program led to an increase of 1.45 percentage points in overall test score, 1.82 percentage points in science scores, and 1.32 percentage points in English scores. At 1.88 percentage points, the relative increase in math scores is also significant at the 10 percent level.

Summing up the findings, we can conclude that SBM in the Philippines had a statistically significant impact on overall student test scores and test scores in English and science. The effect sizes were small to moderate: .10 for overall scores, .13 for science, .09 for English, and .07 for math.<sup>18</sup> These are not inconsistent with Borman’s (2002) analysis, which shows effect sizes of .17 to .14 after one and two years of implementation, respectively, and increasing with years of continued implementation. If we use the coefficients based on OLS and Fixed-Effects models, the effect sizes would be estimated to be lower.

# **6.  Conclusions, Limitations, and Implications for the Philippines**

This study of the SBM program in the Philippines shows that school-averaged student performance on national tests improved between 2002-03 and 2004-05 and that the level of improvement was higher for schools involved in SBM for two years compared with schools that had not yet received the intervention or received the intervention later. School-averaged student performance improved in math, science, and English and on the composite score. Improvement for schools that received SBM early was significantly higher in science and English and on composite test scores.

The Philippines has embarked on a nation-wide effort to introduce and implement SBM. While this paper provides an early indication of the usefulness of SBM in a few districts in the country and its promise for other parts of the country, it also highlights an opportunity for introducing a SBM program rollout design that would permit a more rigorous analysis of the contribution of SBM to student outcomes in the

> 18 See Cohen (1988) for a discussion on the magnitude of effect size.  Effect size is calculated by dividing the coefficient for interaction term, SBMxPostintervention, by the standard deviation of the 2002-03 baseline score.

13
Philippines. We discuss the limitations of the study and provide some recommendations for a program of SBM evaluation in the Philippines.

First although the study found significant differences between SBM and non-SBM schools on schoollevel outcomes, the possibility of unmeasured differences influencing outcomes exists. The findings in impact evaluations are critically dependent on the choice of the comparison group. In the context of this study, although a comparison group was identified, whether the group represented a good counterfactual is more difficult to argue. SBM funds were disbursed in batches and since we had data for all schools, we chose the first batch of SBM schools as the treatment school and created a counterfactual using available administrative data.

Although our model controls for a number of important school-level factors such as student-teacher ratio, school size, school type, and school head type, it is quite possible that other unobservable variables affected the outcomes. For example, the model does not capture student-level data (e.g., socio-economic background), which literature shows to have a significant relationship to student achievement. School personnel quality may similarly play a role. Furthermore, although we rely on the assumption that propensity-score matching would have accounted for some unmeasured differences, important differences cannot be ruled out.

While quasi-experimental evaluation methods are an improvement over simple multivariate techniques, they are not a substitute for more rigorous experimental designs. Taking an experimental approach where the treatment group is randomly selected would allow for a direct test of the effects of the intervention. For instance, if those who received SBM in the first batch were randomly selected with control schools that meet the same criteria, a simple differences-in-difference method would have sufficed. It is understandable that in real world conditions, however, an experimental design can be challenging to implement. While it is difficult to identify control groups that will never receive the program benefits, it is possible to implement a pipeline approach and roll out the program in batches, using randomization or a clear assignment mechanism over time, to enable “control groups” to be built in. This would provide program leaders with an opportunity by which to assess the effectiveness of the intervention among beneficiaries and non-beneficiaries who are similar in every other respect. Such a method would be superior to the one we have presented here, where we rely on statistical methods to create a control group for our analysis.

14
Second, the time frame captured in this study is very small, only two years of SBM preparation and implementation. In order to fully assess the impact of SBM, the assumptions regarding how the reforms play out over time and eventually affect student achievement will need to be articulated and examined explicitly. The possibility of a “hawthorne” effect cannot be eliminated, and, potentially, an evaluation with a longer time-frame would permit the identification of longer-term effects. Behrman and King (2008) highlight the risks of a poorly timed evaluation, ranging from finding partial or no impacts, when they in fact would take a longer time to materialize, to the risk of scaling up a poor program, by waiting too long to evaluate. Articulating clear assumptions regarding the reforms would assist with implementing evaluations in a timely fashion.

Third, the study does not examine the processes through which SBM affected school performance. Barrera-Osorio et al.’s (2009) review of the SBM literature conceptualizes the range of SBM approaches along two dimensions: _who_ has the power to make decisions and the _degree_ of decision-making devolved to the school level. The review notes that “With so many possible combinations of these two dimensions, almost every SBM reform is unique.” (pg. 4). While some programs transfer authority to principals or teachers only, others encourage or mandate parental and community participation, often through school committees. Furthermore, the contexts in which the authority is devolved are likely to play a role in the eventual effects. For example, engaging parents and the wider community in school matters may not be easy in certain contexts – studies indicate that in communities with social and political tensions, the school board may be used as an instrument by the elite to orchestrate greater inequities.<sup>19</sup> Evidence from New Zealand and Australia demonstrates the under-representation of minority groups in the composition of school boards<sup>20</sup> .

Several previous papers have examined some components of, and the processes through which, SBM affects student achievement. For example, Jimenez and Sawada (1998) examined 1996 data on Community Managed Schools (EDUCO), a program created to expand coverage in rural El Salvador, where communities received significant authority over schools, including in financial and staffing areas. They found that enhanced community and parental involvement in EDUCO schools improved students' language skills and diminished student absences, which may have long-term effects on achievement. Sawada and Ragatz (2005) also evaluated the EDUCO program and found that EDUCO transferred few administrative processes to local levels but gave local actors greater perceived influence in hiring and firing teachers. Gunnarsson et al.’s (2004) model of education decentralization suggests that principals,

> 19 Caldwell, 2005.

> 20 Ibid.

15
teachers, and parents are more informed than are national decision-makers in the areas that should be a priority for improving learning quality. Thus, by empowering local authorities to make spending decisions, resources will be allocated more strategically for better quality. Eskeland and Filmer (2002) similarly find that the autonomy of teachers, principals, and parents to make organizational and pedagogical decisions and the participation of parents in schools significantly increased primary school test scores in Argentina. Available reports on TEEP indicate that several aspects of SBM may have been conducive to improved student performance: improved school management through intensive principal and head-teacher training, identification of school-level needs and allocation of resources to those specific needs, greater community attention to schools’ and students’concerns.

Fourth, the study does not examine the distribution of effects across different types of schools, both in terms of specified outcomes – student achievement – as well as unintended effects, such as toll on the principal’s and teachers’ time in community engagement. For example, Berlinski and Galiani (2007) find that Argentina’s decentralization of secondary schools significantly increased test scores overall but decreased scores for schools in poor areas and in provinces with pre-decentralization fiscal deficits. On the other hand, King and Ozler’s (1998) study of School Autonomy, a Nicaraguan SBM program, found that autonomous schools, most of which were catering to deprived areas, yielded results comparable to those of other schools. Unintended effects, or processes that eventually undermine desired effects, may exist as well. The Implementation Completion Report for the TEEP project indicates that, indeed, community engagement demanded that principals and teachers spend considerable time on community relations in addition to their administrative and pedagogical responsibilities, a commitment that several were beginning to find burdensome.

Thus, answers to questions regarding the conditions under which the different SBM models work and which implementation processes are effective are critically important from a policy and program design and implementation perspective. A new evaluation would present an opportunity to study such issues through focusing on such issues as well.

Despite these limitations, this study intends to contribute to the growing body of studies that examine school based management as a tool for improving student outcomes. This study provides an initial case study on the Philippines, an area where SBM has not yet been evaluated. Thus, this analysis suggests there are possible benefits from SBM that may be applicable in the country as a whole.

16
We end this paper with some words of advice for future evaluations of the ongoing SBM program in the Philippines. Barrera-Osorio et al.’s (2009) review of SBM studies concludes that retrospective evaluations of such program are extremely difficult to implement and recommends using prospective methods. The SBM program in the Philippines is an ongoing exercise, and the analysis in our paper suggests that it was successful with respect to its objectives of improving student achievement, although the effects were small. However, as the Department of Education in the Philippines moves forward with SBM and goes to scale across the country, it is important to collect data simultaneously in a systematic manner to enable scientific evaluations. The current data has several limitations which make it difficult to espouse conclusive statements about the success of the program and how it might generalize beyond the 23 districts to the country as a whole. A new evaluation approach could enable the Department of Education to study both the implementation processes (what really makes SBM work) and assess impact in a more rigorous fashion.

17