metadata: authors: 0: author: Alishjabana, Armida
1: author: Artha,Rima Prama
2: author: Beatty,Amanda
3: author: Gaduh,Arya Budhiastra
4: author: Pradhan,Menno Prasad
5: author: Suryadarma,Daniel C. J.
6: author: Wong,Maisy
count: Indonesia
docna: 0: docna: Improving educational quality through
            enhancing community participation : results from a
            randomized field experiment in Indonesia
docty: Policy Research Working Paper
subtopic: Educational Sciences,Educational Institutions & Facilities,Educational Populations,Social Development & Poverty,Education for Development (superceded),Effective Schools and Teachers,Education For All
teratopic: Social Protections and Labor,Education
abstracts: cdata!: This study evaluates the effect of four
            randomized interventions aimed at strengthening school
            committees, and subsequently improving learning outcomes, in
            public primary schools in Indonesia. All study schools were
            randomly allocated to either a control group receiving no
            intervention, or to treatment groups receiving a grant plus
            one or a combination of three interventions: training for
            school committee members, a democratic election of school
            committee members, or facilitated collaboration between the
            school committee and the village council, also called
            linkage. Nearly two years after implementation, the study
            finds that measures to reinforce existing school committee
            structures, the grant and training interventions,
            demonstrate limited or no effects; while measures that
            foster outside ties between the school committee and other
            parties, linkage and election, lead to greater engagement by
            education stakeholders and in turn to learning. Test scores
            improve in Indonesian by 0.17 standard deviations for
            linkage and 0.22 standard deviations for linkage+election.
            The election intervention alone leads to changes in time
            household members accompany children studying per week, but
            this does not lead to learning. Linkage is the most cost
            effective intervention, causing a 0.13 change in standard
            deviation in Indonesian test scores for each 100 dollars
            (US) spent.
geo_reg_key: 517198,517191
repnme: repnme: Improving educational quality through
            enhancing community participation : results from a
            randomized field experiment in Indonesia
display_title: Improving educational quality
            through enhancing community participation : results from a
            randomized field experiment in Indonesia
listing_relative_url: /research/2011/09/15034996/improving-educational-quality-through-enhancing-community-participation-results-randomized-field-experiment-indonesia
topicv3: Teachers Management m1326119 1523,Quality of Education m1326344 491,Education m1327048 470,Social Development m1327660 134,School-Based Management m1327730 2143
publishtoextweb_dt: 2011-09-01T00:00:00Z
historic_topic: Social Protections and Labor,Education
no_of_pages: 50
author_keys: 0: author_key: 955301
1: author_key: 990910
2: author_key: 752708
3: author_key: 123205
4: author_key: 997808
5: author_key: 748091
model_extractions: WPS5795 Policy Research Working Paper 5795 Improving Educational Quality through Enhancing Community Participation Results from a Randomized Field Experiment in Indonesia _Menno Pradhan Daniel Suryadarma Amanda Beatty Maisy Wong Armida Alishjabana Arya Gaduh Rima Prama Artha_ The World Bank East Asia and Pacific Region Human Development Sector Department September 2011
Policy Research Working Paper 5795

# **Abstract**

This study evaluates the effect of four randomized interventions aimed at strengthening school committees, and subsequently improving learning outcomes, in public primary schools in Indonesia. All study schools were randomly allocated to either a control group receiving no intervention, or to treatment groups receiving a grant plus one or a combination of three interventions: training for school committee members, a democratic election of school committee members, or facilitated collaboration between the school committee and the village council, also called linkage. Nearly two years after implementation, the study finds that measures to reinforce existing school committee structures, the grant and training interventions, demonstrate limited or no effects; while measures that foster outside ties between the school committee and other parties, linkage and election, lead to greater engagement by education stakeholders and in turn to learning. Test scores improve in Indonesian by 0.17 standard deviations for linkage and 0.22 standard deviations for linkage+election. The election intervention alone leads to changes in time household members accompany children studying per week, but this does not lead to learning. Linkage is the most cost effective intervention, causing a 0.13 change in standard deviation in Indonesian test scores for each 100 dollars (US) spent.

This paper is a product of the Human Development Sector Department, East Asia and Pacific Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at amandabeatty@post.harvard.edu, mpradhan@feweb.vu.nl, and daniel.suryadarma@anu.edu.au.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
Improving Educational Quality through Enhancing Community Participation: Results from a Randomized Field Experiment in Indonesia<sup>1</sup> Menno Pradhan<sup></sup> Daniel Suryadarma<sup></sup> Amanda Beatty<sup></sup> Maisy Wong ^ Armida Alishjabana “ Arya Gaduh Rima Prama Artha .

# **JEL Classification: I – Health, Education, and Welfare Keywords: school committee, primary schools, basic education**

1 A large number of people contributed to the design, implementation and supervision of this research project. From the Indonesian Ministry of National Education: Bambang Indriyanto, Sri Renani Pantjastuti, Sri Amien, Dasim Budimansyah, Agus Haryanto, Yadi Haryadi, Neneng Kodri, Suparlan, Anen Tumanggung, Yudistira Widiasana, Diana Sufa, Ismulyanto Apriwibowo. From the World Bank and affiliates: Vivi Alatas, Desmond Ang, Chris Bjork, Esther Duflo, Scott Guggenheim, Djoko Hartono, Dedy Junaedi, Siswantoro, Rosfita Roesli, Chris Thomas, Jeremy Tobacman, Tri Yuwono. We are grateful to Matt Stevens, Deborah Cobb-Clark, Tue Gørgens, Andrew Leigh, Matthew Grant Wai-Poi, Astid Zwager and seminar participants at the Australian National University, Tinbergen Institute and CEPR development economics seminar for helpful comments and suggestions. We acknowledge financial support from the Japan Social Development Fund, and the Dutch Government. The firm PPA and Moores Roland implemented the intervention and survey respectively. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.

>  VU University Amsterdam & University of Amsterdam.mpradhan@feweb.vu.nl. ;# Australian National University;

> * World Bank Indonesia; ^Univeristy of Pennsylvania; Universitas Padjadjaran, Bandung, Indonesia

1
# **I. Introduction**

As educational enrollment and attainment rise worldwide, research has started to show that cognitive ability, rather than attainment or enrollment, strongly contributes to labor market outcomes and economic growth (Hanushek and Woessmann (2008)). This finding is gaining traction in policy circles, with policymakers turning to interventions that address deficiencies in learning.<sup>2</sup> One prominent method of improving education outcomes is to strengthen local school governance structures, reinforced by community contributions. Services work best when they reflect local priorities and meet the needs of the clients they are serving; and when service providers are accountable to clients (World Bank (2003)). Communities that are informed and engaged in education can promote accountability and learning by monitoring education performance, advocating for improved services, and encouraging learning in and out of school (Bruns et al. (2011)).

A school committee or council, comprised of teachers, school leadership, parents and/or community members, is a common mechanism of promoting community involvement in education (F. Barrera et al. (2009)). The committee may take on a variety of school and community functions, such as approving school expenditure, participating in school budgeting processes, monitoring teacher performance, publicizing the status of learning in the community, raising funds for the school, serving as a channel for parental concerns, encouraging parents to monitor and assist in student learning at home, or facilitating communication and cooperation between school management, parents and the community.

This paper examines the impact of several strategies to strengthen school committees with the aim of increasing community involvement in education, accountability of schools and ultimately learning. The study takes place in Indonesia, where in 2002 the Ministry of National Education (MoNE) passed a decree that expanded the role of school committees. Whereas prior to the decree the main role of the school committee was to raise funds from parents and the community, the new law stipulated a broader role in school based management: to advise and support school principals and teachers; to act as a mediator between the school and the community; and to improve parental and community involvement in school activities.

By 2006, several years after the decree was established, the school committees‟ role had been largely unrealized. So this pilot study was designed to test four measures aimed at helping school committees fulfill the role envisioned in the decree, and eventually improve student learning. The measures were hoped to increase the capacity and knowledge of the school committees (through training and financial resources), promote community representation on the committee (through democratic elections), and foster ties between the school committee and a local governing body thus promoting the committee‟s authority (called linkage). To our knowledge, the election and linkage interventions have not yet been rigorously evaluated elsewhere.

2 See for example, the World Bank‟s Education Strategy 2020 www.worldbank.org/education. 2
As part of this study, school committees were randomly allocated to receive a grant, a grant in combination with training and/or election and/or linkage, or no intervention, which served as a control. Thus, the evaluation uniquely tests eight different combinations of interventions in the area of school based management and community engagement against a comparison group. The study took place in 520 randomly selected rural public schools in six districts in Central Java and Yogyakarta, Indonesia, over a two year period starting in 2007.

We find that interventions that reinforce existing school committee structures – grant and training – demonstrate limited impact on learning; while those that foster ties between the school committee and outside parties – linkage and election – are successful. Two years after the start of the project, linkage, and linkage+elections, conditional on receiving a grant, are the only interventions that show a positive impact on learning. Indonesian test scores increase by 0.17 standard deviations for linkage and 0.22 standard deviations for linkage+elections. For math, the linkage intervention has a 0.11 standard deviation effect for girls.

Placing the Indonesia results in context with other studies attempt to strengthen local school governance bodies, we find some parallels and departures.<sup>3</sup> In the Indonesia case, grants to committees help raise parents‟ awareness about the existence of the school committee and its members, and increase internal school committee meetings; but do not yield learning results. Training, the most costly intervention, has the smallest impacts on the intermediate outcome variables. The evaluation most similar to our context is Blimpo and Evans (2010) in the Gambia, where they are also able to separate effects of grant versus training, or rather grant alone and grant plus training. While neither intervention shows an effect on learning, with the grant plus training intervention, there is a change in grade 1 enrollment, and other aspects of school-based management, such as establishing school management committees.

Looking at evaluations that show more promising results for grant and training interventions, some show changes in learning, or at least changes in repetition, failure, and enrollment rates – none of which change in this evaluation. Gertler et al. (2008) find in Mexico that a decline in repetition and failure rates of five percent, with no effect on dropout rates, as a result of grants amounting to $500-700 and training to parent associations. The later Gertler et al. (2010) evaluation in the same context double grant amounts, showing an impact on the dropout rate, and Spanish and math learning (five to eight percent increase in test scores), yet no effect on failure or repetition rates. Khattri et al. (2010) also show 1.5 percentage point difference in test scores from a combination of grants and training offered to principals and head teachers in the Philippines. In another example of fortifying a school committee, Duflo et al (2009) show results from providing a committee with funding for contract teachers and training in monitoring their performance. Learning effects

> 3 In these studies, some kind of school committee, comprised of teachers, principals, parents and/or community members, was provided with training, and money to be used for school-level projects, usually infrastructure or materials, but occasionally also locally-hired teachers. 3
are largest in schools in which school committees are trained (0.21 change in standard deviations in math test scores), suggesting that one mechanism of improved learning was through additional teacher effort resulting from committee monitoring.

As mentioned above, in our Indonesia example, interventions that foster outside ties between the school committee and other parties – linkage and election – are more successful than the grant and training interventions, yet have no direct international comparisons.<sup>4</sup> Thus we draw on the literature related to mobilizing communities to enhance service provision. The distinction with the stakeholder engagement evaluations is that they are generally not targeted at strengthening a provider-level body, as in our case the school committee, but rather providing training and information to the community, such as parents. We find the community engagement and information literature most relevant because the election intervention was designed to broaden participation to groups that hadn‟t been represented before, while the linkage intervention was intended to strengthen relationships between the school committee and important education stakeholders in the community – members of the village council.

Analysis of intermediate outcome variables in the Indonesia experiment reveals that the linkage intervention indeed leads to increased collaboration between the school and the village council. This additional support from the community for the school is confirmed by parents. However, although linkage was designed to enhance the role of the school committee, quantitative, complementary qualitative analysis by Bjork (2009) indicates that the collaboration was primarily between school management and the village council, with a marginal role for the school committee. This suggests that linkage has an effect on learning bypassing the school committee, rather than by empowering it, at as originally intended. In addition to improving learning when combined with linkage, elections appear to have broadened community awareness and action. They raise overall awareness about the school committee, engender respect for the school committee in the eyes of the teachers, increase time household members help their children with homework, and prompt greater effort by teachers, largely spent outside the classroom.

Other studies also show a relationship between different varieties of stakeholder engagement and learning. Nguyen and Lassibille (2008) find in Madagascar that providing parents with school report cards and information about what they can do to improve the quality of education, and offering pedagogical and administrative tools to teachers, prompts a 0.1 standard deviation increase in test scores and improvements in student attendance. Banerjee et al. (2010) find that recruiting and training community volunteers in how to

> 4 One might make the parallel with Banerjee et al. (2010), which tests interventions that were hoped to energize a village education council (VEC) that includes the head of the village council. But this experiment did not specifically target building relationships between village government and the school committee. In fact, it hoped to embolden community engagement as a counterbalance to village government‟s strong influence on the VEC. This counterbalance did not materialize as a result of information sessions about the role of the VEC and its responsibilities. 4
teach the low-performing students reading improves learning, while disseminating information to communities about rights and responsibilities regarding education and the state of learning in the village does not (although it does improve general awareness about education). In another promising example of local-level engagement, Bjorkman and Svensson (2009) show improvements in service utilization, quality of service delivery and health outcomes in Uganda, as a result of organized meetings with community members and health service providers about the level of services in the village, and provider responsibilities.

In the following section we discuss the motivation for the field experiment, and describe the interventions in detail. Section III outlines the sampling strategy, timing and what information was collected. We then present the approach used in the empirical analysis (section IV), followed by the results (section V). Section VI discusses results from a companion qualitative study conducted in a subset of the schools, and Section VII outlines what factors may have led to learning in the linkage and linkage+election interventions. We conclude in section VIII with a summary of findings.

# **II. Motivation, intervention design and implementation**

After achieving universal primary school enrollment in the 1980s (current gross enrollment is 110 percent), Indonesia began to shift attention to quality with reforms such as teacher training and upgrading, curricula revision, facility improvements, and later on school-based management (Kristiansen, Stein and Pratikno (2006)).<sup>5</sup> Despite these initiatives, Indonesia awaits marked progress in learning. In reading, it ranks 57<sup>th</sup> out of 65 countries that participated in the Program for International Student Assessment (PISA) in 2009 (OECED (2009)); and in the Trends in International Mathematics and Science Study (TIMSS) 2007 (I.V.S. Mullis et al. (2008)), only half of Indonesia‟s students performed above the lowest international benchmark. Hanushek (2008) also finds that just nearly 30 percent of a cohort of grade nine Indonesian students had achieved full literacy. Data on service quality suggest substantial scope for improvements in efficiency. Teacher absenteeism is estimated at 15 percent (SMERU (2008)).

Against this backdrop of national efforts at promoting education quality, the Government of Indonesia in 2002 instituted a decree that gave school committees a greater role in advising and supporting school management, and encouraged greater engagement with community.<sup>6</sup> The decree stipulated that school committees would replace existing school-level committees known as BP3 ( _badan pembantu penyelenggaraan pendidikan_ ) in Indonesian. The primary function of the BP3 was to raise funds from parents and the community to support the school, yet the funds were largely handed over to principals. The school committee would go a few steps further, making recommendations on school expenditures, teacher

> 5 Enrollment figures calculated using SUSENAS 2009. Enrollments dipped below 100 percent during the 1997-98 crisis, but returned to pre-crisis levels in the 2000s. 6 Lampiran I Keputusan Menteri Pendidikan Nasional,Nomor 044/U/2002, Tanggal 2 April 2002 5
qualifications, and school facilities. In addition, the school committee was expected to act as a mediator between the school and the community, and promote community, especially parental, involvement in the school.<sup>7</sup> Although the decree had been passed in 2002, four years later the decree had had limited effect on the actual functioning of school committees – they were largely still operating under the BP3 model (Fearnley-Sander et al. (2008). This result begged the question of what could be done to help school committees realize the role envisioned in the decree, yet were cost-effective and scalable. Field visits and further discussions with the Ministry regarding lessons learned from other education projects led to the development of four approaches tested as part of this experiment. Here we discuss the motivation for choosing each of the interventions, and how they were implemented in practice.<sup>8</sup> This experiment was funded by a grant from the Japanese government to the Ministry, and the Ministry contracted out intervention implementation to consulting firm, Pusat Pengembangan Agribisnis (PPA),<sup>9</sup> for a total contract value of 2.9 billion Rupiah (US$315,000).

# _Intervention 1: Grant and facilitation_

All 420 treatment school committees received a block grant of eight million Rupiah (US$870) with the expectation that the grant would complement the three other treatments described below, especially training and linkage. The grant had three objectives. First, it would allow school committees to greater contribute to school activities, since traditionally school committees did not have access to resources. Second, the grant would allow the school committee to practice budgeting and planning, which members had little experience with previously, and would also be covered in the training (see more below). Third, the grant would allow the committee to reach out more easily to parents, community members and school management because they had money to hold meetings, and planning how to spend the grant was an occasion for meeting. Grant expenditure planning was also to be done with the village council as part of the linkage treatment.

The school committee did not receive the money without strings, but rather was expected to develop (together with the village council for schools that received the linkage treatment, see below) a plan for expenditure, and the committee was required to be transparent by posting expenditure categories on the

> 7 In Indonesia, districts and the central government are largely responsible for teacher employment and deployment. At the school level, the headmasters and sometimes school committees would only have the authority to hire or fire contract teachers.

> 8 Compliance with intervention assignment was relatively good, with the exception of the election intervention (see text), and two schools: one school refused to participate at all at the outset, and another school refused to participate after the elections caused a conflict with the existing school committee.

> 9 The World Bank supported PPA by making available a consultant for about two months, with the task of assisting in planning the interventions.

6
school notice board. The school committee developed an expenditure plan with the assistance of the facilitators supported by the project, who coached school committees on how they might address problems at the school with the block grant (but only those that could be implemented in two years, or the life of the experiment), approved expenditure proposals from school committees, authorized transfer of the block grant (once they approved expenditure proposals), ensured transfer of the grant to school committees‟ bank accounts, and monitored the use of the block grant. On average, one facilitator was assigned to ten schools, and visited each school committee 13 times.<sup>10</sup> Using these estimates on the number of visits, time facilitators spent in the schools, and staff salaries to break down the facilitation costs, and considering other treatmentspecific costs, we estimate that the cost of implementing the grant was about US$321 (excluding the grant itself) per school.

The eight million Rupiah (US$870) block grant was transferred directly from the MoNE into a bank account held by the school committee, in two tranches, with the first tranche amounting to three million Rupiah (US$326). This first tranche was disbursed in January 2008, three months later than planned, due to budgeting problems at the MoNE. The second tranche also confronted Government budgeting delays. It was to be disbursed to the schools subject to sufficient progress achieved by the school committee in using the first tranche of the grant; but, in practice, all schools received the second tranche, and received it ten months late, in December 2008, after the endline survey.<sup>11</sup> Thus, these results measure the impact of the first tranche, and the anticipation of getting remaining funding.

# _Intervention 2: Training_

Other factors hypothesized to be holding back school communities from realizing their role was information, such as their lack of knowledge about the decree; and capacity, such as how to engage the community, how to play a role in school management, and how to promote student learning. Thus, a two-day, district-level training attended by four school committee members (principal, teacher, parent, and one village representative)<sup>12</sup> covered planning, budgeting and steps the school committee could take to support education quality. The budget session focused on a plan for spending the block grant. Materials drew heavily on the Creating Learning Communities for Children (CLCC) model developed by UNICEF,<sup>13</sup> which provides prolonged training and facilitation to schools on active learning, school-based management and

> 10 Based on interviews by the authors with two district facilitators after completion of the project.  According to PPA, the project employed 50 facilitators for a period of 15 months, and six district facilitators who managed the district teams. The interventions were implemented consecutively – elections, linkage, and then training. 11 The endline survey had to be conducted before the second tranche was disbursed because the grant from the Japan Social Development Fund that financed the survey was about to expire.

> 12 For the schools that also received the linkage treatment, one additional representative from the village council was invited.

13 For more on Creating Learning Communities for Children, see http://www.unicef.org/indonesia/resources_7230.html. 7
community participation, and has served as the foundation for several donor projects promoting school-based management in Indonesia. Naturally, it was not replicated fully for this project because the cost per school for such intensive work would have been too high. The training also included a visit to a „model‟ school committee that had been successful in applying school based management practices.

Schools receiving the training intervention received three additional visits beyond those provided with the grant. One visit was to announce the training and to agree on who would be participate; another to deliver an official invitation stamped by the district education office; and a final visit took place just before the training, to ensure that those invited would come. Implementing the training cost US$360 (including cost of training) per school.

# _Intervention 3: Election_

The primary concern to be addressed by the election intervention was that school committee members were often handpicked by school management and did not represent parents or the broader community. With a democratic mandate and greater diversity in membership, it was hoped that the school committee would better communicate with parents and community members, and act in their interest with more authority and voice.

The intervention introduced two primary changes to the process outlined in the decree: a quota for different types of members rather than a minimum or maximum number of members, and election committee‟s role as facilitator for elections rather than an election body.<sup>14</sup> The 2002 decree stipulates that the school committee include at least nine members, including community representatives (maximum of three from the village government), teachers, parents and the principal (although he/she cannot be the head); and the community must propose candidates. The intervention tightened these guidelines, designating that the committee be comprised of six parents, three community members, one teacher, the principal and the head of the village council. The rationale for the quota was that it would ensure greater parental representation, and allow parent and community stakeholder groups to directly choose their own representatives.

The intervention also redefined the role of the election committee and gave more structure to the election process. The 2002 decree actually requires an election and envisions that members be elected by a voting election committee, which in practice happened; but the committee was often chosen by principals, and candidates did not represent a variety of education stakeholders. The intervention instead delegated the

> 14 The election process was modeled after that used in the Urban Poverty Project (UPP), known by the Indonesian acronym P2KP. The experience of UPP pilots, in which membership of village government was put to a popular vote, was that elites stood for election and were elected. Thus, the project modified election processes to mobilize candidates from different sectors of the community, which is the model used in this experiment. 8
election committee responsibility for mobilizing voters and holding the election. Facilitators from the project socialized<sup>15</sup> schools and communities about the new process and assisted the two stakeholder groups, parents and community members, in selecting an election committee.<sup>16</sup> Once the election committees were established, the facilitators, along with election committee members, undertook human resource mapping for the community and parental groups,<sup>17</sup> which led to candidate selection. After candidates were proposed, the community and parents groups elected their members.<sup>18</sup> Subsequent meetings were held to sign a decree establishing the committee and to develop a workplan. The election process generally took facilitators five visits to schools beyond those necessitated by grant implementation, and took place in three batches to spread out facilitator workloads.<sup>19</sup> The intervention cost approximately US $174 per school.

Despite the efforts of the implementer PPA to encourage communities to remain faithful to the design outlined above, some schools refused to conduct an election. As shown in Table 3, only 48 percent of the schools randomly selected to implement an election actually did as intended. Of those schools that didn‟t comply, about seven percent of committees refused to change any members, while the remainder of committees agreed to a compromise of electing representatives who were members of previously unrepresented groups. Some of those that refused or partially refused claimed that some school committees were only starting their terms, and thus they did not want to start over with new membership after a new pool had just been appointed or elected. Despite this partial noncompliance, elections indeed resulted in a higher share of new members being elected to school committees, as shown in Figure 2. The number of members in the school committee increased by about two as a result of the election, and the share of parent representatives increased. In the majority of the schools receiving this intervention, more than half of the school committee members started their terms between baseline and endline, whereas in schools that did not receive the election intervention, hardly any new members joined the school committee during the course of this study.

_Intervention 4: Linkage_

> 15 Socialized in this context means spreading information about the project.

> 16 The election committee is selected through two separate but similar processes at the school and village levels. At the school, two parents and one teacher are selected based on a plurality of votes cast by the principal and teachers. The teacher selected in this process automatically becomes a member of the school committee. Similarly, at the village level, two individuals elected during a village-level meeting are designated to the election committee.

> 17 The village human resource mapping involved village organizations and community leaders, and allowed groups to propose potential candidates as community representatives.  Representatives at this meeting recommended five people per organization or group as potential candidates, who were then invited at the next community meeting.  A similar, separate meeting was held for parents of children in grades one to five, where the desired qualifications of school committee members were discussed, and potential candidates suggested.

> 18 Instead of a ballot, voters wrote down names of candidates, and the ones with the highest votes won.

> 19 Batches took place 15 April to 31 May 2007. 1 June to 14 July 2007. 15 July to 31 August 2007.

9
Another concern raised by the Ministry and committee representatives was that school committees had no formal power to hold the school management accountable. Providing formal power to school committees as part of the experiment would have meant revising the decree or changing authority structures at the district level, which was not feasible at the time of design. But the design team thought it possible to implement a mechanism for providing school committees access to informal mechanisms of voicing concerns, raising the status of the school committees in the village, and better enabling committees to mobilize community support. Collaboration with a powerful local body -- the village council (known in Indonesian as _Badan Perwakilan Desa_ or BPD) -- was that mechanism. The intervention represents the spirit of the decree since the decree even envisions village representation in the school committee; but findings from field visits indicated, and baseline data confirmed, that there was little evidence of this collaboration. At baseline, 22 percent of all school committee representatives reported collaboration with the village council.

It was hypothesized that linking the school committee with the village council would first increase the stature of the school committee vis-à-vis school management, improving the ability of the school committee to exert more influence to improve services; and second generate support for the school in addressing communitylevel problems that could not be solved by school management alone. The intervention cost US $125 per school, mainly covering the two additional visits to the school beyond those provided with the grant. The first facilitated meeting was between school principal and school committee members to identify measures for improving education quality that they would then propose to the village council. These measures were discussed in a subsequent meeting with village council representatives and other village officials, and the results of the meeting were documented in a memorandum of understanding signed by the head of the school committee, the head of the village council, and the school principal. Examples of measures that parties collaborated on included building school facilities, establishing village study hours (two hours in the evening when households would turn off televisions and computer game kiosks would be closed), hiring contract teachers, making land available for school infrastructure expansion, resolving conflicts between two schools in a community and encouraging social and religious activities at school. In some cases, collaboration even extended to village council representatives becoming school committees members (Christopher Bjork (2009)).

# **III. Sample, timeline and survey content**

This study took place in nine districts in central Java and Yogyakarta, a region chosen because there were few large education projects active in the area, enabling the results to be relatively free from the risk of contamination from other projects. Moreover, conditions were hypothesized to be ripe for community engagement to flourish – the area is peaceful, has reasonably high levels of existing social capital, and schools are relatively well-equipped (high levels of electricity, adequate number of teachers, etc.). The 10
evaluation also focuses on public primary rural schools – public because this evaluation was designed by the Ministry, which has the authority over public schools,<sup>20</sup> and rural because the majority of schools in the country are in rural or semi-rural areas, and it was hypothesized that accountability would be easier to engender in smaller, closer-knit areas.

So, from two provinces, nine districts and primary schools, the sampling frame was further restricted by excluding subdistricts containing fewer than eight villages,<sup>21</sup> schools with extremely good or bad average sixth grade examination scores in mathematics or Indonesian,<sup>22</sup> and schools with parallel classes in grade four.<sup>23</sup> We used a two stage sampling procedure, first drawing 44 subdistricts and then 520 schools. Each potential sample school in the pool had an equal probability of being sampled. No more than one school was drawn for each village, to reduce the risk of spill over between interventions.<sup>24</sup> In drawing the school sample, we ensured that the sample was balanced with schools with low, medium and high grade six leaving exam scores across interventions in each cell. The resulting sample of 520 schools was randomly allocated to interventions according to Table 1.<sup>25</sup> A disproportionate share of the sample was allocated to the cells receiving nothing or just the block grant, in order to be able to separately identify the effect of the block grant. The training intervention was made slightly smaller than the non-training cells because training is a relatively costly intervention.

The baseline survey took place in January 2007, midline in April 2008, and the endline survey in October 2008, as shown in Table 2.<sup>26</sup> Tests in mathematics and Indonesian, designed by MoNE, were administered to all students in grade four at baseline and six at endline.<sup>27</sup> Survey questions center around hypothesized

> 20 MoNE supplies some support to private schools, but has direct oversight over public schools.

> 21 This restriction was imposed because in the initial design, facilitated meetings with sub-district government education officials were envisioned, and too few villages per meeting would make this intervention ineffective. However, this idea was never implemented, making the restriction unnecessary.

> 22 That meant those with the grade six average student scores below four or above eight. School-level score data obtained from MoNE.

> 23 Parallel classes are grades with more than one section or teacher. This restriction was imposed because the evaluation was not planning on assigning student IDs or ensuring that the student population was identical over time. With only one class per grade, and low dropout and repetition rates, the evaluation team was confident that the same children interviewed in grade four would be in grade six two years later. However, this actually became an issue, since several schools merged, but the team was able to match student names, see below.

> 24 The sampling probability was increased accordingly for schools that were located in villages with more than one school to keep the probability of being sampled equal across schools.

> 25 Random allocation conducted by authors.

26 The survey was conducted by a Moores Roland, a survey firm by MoNE. Five out of 520 schools were not included in the endline survey. Two schools merged, and one school refused interview. In two other schools, the implementer implemented the treatment in schools different than those surveyed in the baseline, so the team felt it unnecessary to survey non-randomly selected schools for which there was no baseline information.. In addition, there were some difficulties with non-response in some of the schools, so we have complete information on 508 schools and partial information on seven schools.

> 27 In the original sample selection, schools with multiple parallel classes were excluded (so all sample schools started out with one grade 4 class), and thus we did not assign student IDs; but in the endline survey, it was discovered that several schools had more than one grade six class, due to schools merging.  We only have data on the number of

11
intermediate steps along a path from school committee actions to improved learning outcomes (see Figure 1). Broadly, these relate to community support for education, parental support for learning, school based management, and teacher opinions and behavior, so we interviewed parents, teachers, students, school committee members, and principals. To track the teachers of the students tested, the teacher sample was restricted to teachers teaching grade four at baseline and grade six at endline. We then randomly selected three students from their classes, and these students‟ parents, for interview.

# **IV. Impact evaluation strategy**

In this section we discuss the analytical approach we use to designate the appropriate comparison group given the combinations of interventions, determine test results, handle a lack of compliance with the election intervention, generate results using a large number of outcome variables, and check for baseline imbalances across treatment and control groups.

# _Pairwise impact evaluation_

As discussed above, the objective of this study is to evaluate the effects of four interventions, independently and combined with each other. To estimate this, we analyze seven pairwise comparisons found in Table 4. The grant comparison examines the schools in the control group with those that received the grant only, while all other comparisons measure the effect of the other interventions (election, linkage and training) and their combinations, conditional on receiving the grant. Note that the sample size for the three combination comparisons is smaller compared to the others with grant plus only one other intervention, as we exclude the schools that received only election, training or linkage interventions.

We apply weights equal to one over the number of observations in the cell (see Table 1) to ensure that the comparison is balanced across the interventions that are not studied. For instance, when we compare the effect of training, the weighting ensures that the two groups are equally exposed to the linkage and election interventions. Randomized trials in general ensure that in expectation the control and treatment groups are equally exposed to other effects which are not controlled by the experiment. Our strategy ensures that this is also the case for the non-studied interventions in the comparison.

# _Impact on test scores_

The impact of the intervention on test scores is estimated by parallel grade six classes in 240 sample schools, and found that 13 grade six classes had two parallel classes (5 percent). However this issue is remedied by matching student names, see below.

12
> where _yi_ , _j_ denotes the standardized test score of student _i_ in school _j_ in strata _k_ . The standardized test scores are calculated by subtracting the mean and dividing the test score by the standard deviation observed in the control group schools. Note that the baseline value for _y_ observes the students in grade 4, while by endline these students were in grade 6.<sup>28</sup> The results are calculated using a balanced panel. Standard errors were calculated allowing for correlation between the error terms at the school level. The treatment variable equals 1 for the treatment group, and 0 for the comparison group.

# _Intention-to-treat with the election intervention_

As mentioned above, about half of the schools receiving the election treatment refused to hold an election for all members, so for this intervention we present both the intent to treat (ITT) and the instrumental variable (IV) estimates of the local average treatment effect. For the former, all schools that _received_ the election intervention (irrespective of whether it was fully implemented), the treatment variable is set equal to 1. The ITT effect estimate measures the results of the election intervention as it was actually implemented, with some committees only partially elected. For the latter, the IV estimate of impact of the election, we create a new treatment variable: the share of members that are new to the school committee between baseline and endline. This variable indicates the intensity by which the election was implemented, accounting for the possibility of partial elections, and is highly correlated with the initial assignment (see Figure 2). IV extrapolates findings to estimate the effect as if the full school committee had been elected. The treatment effect is estimated by instrumental variables, where the new treatment variable is instrumented with the initial assignment. Not surprisingly, where we find election effects, the effects are generally larger for the IV versus ITT estimates.

# _Impact on intermediate outcomes_

We not only wanted to understand effects on student learning, but also changes in other variables that were hypothesized to be precursors to learning improvements. Because we collected wide variety of data at the school and household levels (over 100 variables), our challenge was to pare down the analysis to a suite of outcomes that made intuitive sense as pathways to learning, instead of only choosing to report those that

> 28 Student test results were matched based on the names of the students written on the test sheets. This resulted in 11,463 tests/students that could be matched, which is equal to 90 percent of the tests administered at baseline, and 92 percent of the tests administered at endline. Only matched data were used in the impact estimations. The students‟ gender variable was also constructed ex-post using the name of the student. 13
showed empirical results. Following Kling et al. (2007) and Banerjee et al. (2010), , we construct a summary index for each domain. We define the summary index score for school _j_ over the set of _N D_ outcome variables in group _D_ as the mean of the z scores outcome variables in a group. Our strategy was to define domains corresponding to the logical framework as presented in Figure 1, and construct variables (that feed into summary indexes and then domains) accordingly. Each variable is constructed such that it contributes positively to the header or overall concept used for the domain. Often information on the same topic in the same summary index is obtained from different respondents.

Where<sup>_~~y~~_</sup> _d_<sup>group</sup>  _d_ are the mean and standard deviations of variable _y jd_ estimated from the control group schools. The summary index provides an equal weight to each variable that enters the summary index correcting for natural variation as observed in the control group.

For the intermediate outcome variables and the summary indices we apply a similar estimation method, only we condition on the baseline values of all variables included in the summary index. For instance, for outcome variable _d_ included in summary index _D,_ the impact is estimated by For estimating the impact on the summary index itself , _yi,d,endline_ is replaced by _yi,D,endline_ .

Using the technical strategy above, we intuitively group intermediate outcome variables according to the impact domain they refer to. Each impact domain is represented by a table (see Tables 8, and 10-19 below). The list of variables in each domain, their definitions and the source of the data (i.e., corresponding questionnaire) is found in Table 5. In order to further guide the discussion (see results section V on intermediate outcomes), we have grouped impact domains into subject themes pertaining to similar intermediate outcomes. This grouping is as follows

1. Results related to education outcomes

- a. Stakeholder satisfaction with student learning (Table 8)

2. School committee and community support for education

- a. The number of school committee meetings with education stakeholders (Table 10)

- b. Stakeholder opinions about school committee effectiveness (Table 11)

14
- c. Village councils‟ collaboration with schools and overall support for education in the village (Table 12)

- d. Community support for schools and school committees (Table 13)

3. Parents‟ effort, engagement and support

- a. Parents‟ financial and in-kind support to school committees (Table 14)

- b. Parents' awareness of school committees (Table 15)

- c. Parents' support for and involvement in education (Table 16)

4. School-based management and accountability

- a. Number of teachers (Table 17)

- b. Financial accountability of school management to parents and school committees (Table 18)

- c. Principals‟ performance and management of teachers (Table 19)

5. Teacher motivation and effort

- a. Teacher motivation and effort (Table 20).

# _Checking baseline, pre-treatment differences_

In order to check whether our baseline outcome values are balanced across intervention groups, we apply a strategy similar to that discussed in the summary index section above. In Table 6, we report the estimates of coefficient of  in (1) and (3) where the dependent variable is replaced by the baseline value and only the treatment variable is included on the right hand side. Naturally for the election, only the intent to treat comparison is shown. To save space, we present the results for the language and mathematics test scores and the summary index only. No difference can be found for the baseline test comparisons, and the same holds for most summary measures. Focusing on differences with a significance level of 5 percent or below, we find that the treatment groups in the linkage and linkage+election schools have stakeholders with a higher opinion about school committee effectiveness (measure 1), and fewer teachers (measure 7) at baseline.

# **V. Results**

In this section we discuss the impacts on education outcomes, and also on the intermediate outcome variables that were hypothesized to lead to learning.

# **Education outcomes (Tables 7 and 8)**

We begin by measuring the impact of the interventions on the main education outcome variables –- dropout rate, repetition rate, and test scores, which are shown in Table 7. We do not find any significant effects on dropout and repetition rates, which is not surprising given the very low rates at baseline (0.004 and 0.024, 15
respectively). The lack of effects on dropout and repetition rates makes us more confident about comparing test scores of grade 4 and grade 6 students, since if there had been effects, we would have been worried about endogenous attrition causing sample selection bias in our results.

Looking at learning, we find substantial effects in Indonesian, and no effects in mathematics for the two pairwise comparisons shown in Table 4. Linkage improves Indonesian learning by 0.17 standard deviations, while the linkage+election increases Indonesian test scores by 0.22 standard deviations (ITT/0.38 for IV).

The effects of linkage and linkage+election are larger for children with high baseline scores, specifically the highest two ability quintiles, and for girls. The fourth and fifth quintiles saw a respective 0.18 and 0.37 standard deviation gain in Indonesian with linkage+election, and a 0.24 standard deviation gain with linkage. Girls were already the better performers at baseline. Whereas the lowest scoring quintile at baseline is made up of 52 percent boys, the highest scoring quintile has only 38 percent boys. For math, the only significant effect is with the linkage intervention for girls, for who test scores increase by 0.11 standard deviations.

In addition to effects in linkage and linkage+election interventions, the quintile analysis also shows effects for the grant intervention in Indonesian, albeit only in the highest ability quintile (0.3 standard deviations), and for the linkage+training intervention with an increase of 0.24 standard deviations for the fourth quintile.

We suspect that the lack of significant results in mathematics is in part caused by the test design. Figure 3 shows a histogram of the scores for all the tests. The maximum number of points that could be obtained was 30 for each test. The endline mathematics test in particular seems to have been rather difficult. A very large proportion of the students scored around 7 points. This suggests that the test was not as good at demonstrating a range of abilities as was the Indonesian test. Figure 3 also shows that there is no evidence of ceiling or floor effects for the math scores – in both years the histograms have the expected bell shape curves. The same is true for Indonesian scores at baseline, but the endline histogram has two humps, where the right hump is cut by the maximum score. There is however no clear excess mass at the maximum score, suggesting limited truncation. The two bells found in the curve are in line with the finding that the strongest effects were found for the higher scoring students.

While these improvements in learning are substantial, we do not see improvements corroborated by education stakeholder respondents. Table 8 presents the subjective assessments with students learning of school committee representatives, principals, teachers and parents,<sup>29</sup> who do not demonstrate significant changes in satisfaction with learning for any intervention other than grant (decline in principals‟ satisfaction

> 29 Since so many parents (nearly 20 percent) responded with don‟t know when asked to rate their satisfaction with their child‟s learning, we changed the variable to a dummy regarding whether parents were able to respond to the question. We see no change in this variable for any intervention. 16
by 7 percentage points). Perhaps such subtle changes in learning were not easily observed by school committee representatives, parents or principals. One might expect teachers to notice this most readily, but note that satisfaction levels are generally reported by different teachers, since grade four teachers were interviewed at baseline and grade six at endline, and a grade six teacher might not be aware of the ability that a cohort of students had in grade four.

# _Cost effectiveness_

Using the costs discussed in the implementation section above,<sup>30</sup> we find that the linkage intervention is the most cost effective – the benefit of spending USD 100 on Indonesian test scores measured in terms of standard deviations is 0.13 for the linkage intervention, and 0.07 for the linkage+election intervention. In these calculations, we consider the costs of the specific intervention<sup>31</sup> plus the facilitation costs associated with the intervention. However, we did not include the cost of general facilitation and the grant since the impact results are conditional on receiving a grant. While it is possible that the linkage, election and training interventions became more effective as a result of the general facilitation and grant being available, this hypothesis is not testable as interventions were implemented simultaneously, or at least closely parallel to each other.

# **Intermediate outcomes (Tables 9-20)**

As discussed above, learning, dropout and repetition rates are our primary outcome indicators. But we don‟t want to just understand _that_ or _why_ learning improved<sup>32</sup> , but also what other variables, other than learning, are affected by the interventions, because improvements in these areas may be goals themselves, and they were hypothesized to ultimately lead to improved learning. Thus, in the following sections, we discuss results for intermediate variables individually and by summary index, mainly relying on the individual variable explanations when we see no effects at the summary level. Recall that every summary index is represented by each of the tables 8 and 10-20.

As a preview of the results, we tabulate the fraction of coefficients that showed a significant effect by respondent group and intervention, at the 5 percent level or below, shown in Table 9. Examining findings by intervention, we find that the grant showed the most traction with variables related to parents, the elections

> 30 See section II for details on the cost of each intervention. The only cost not mentioned in section II above above was overhead costs (team leader, office space, etc) allocated equally to all intervention schools, which amounted to US$ 140 per school.

> 31 We used the intent to treat results for the election benefits. The costs, which covered facilitator vists, would likely have been higher had a full election of the school committee been enforced, necessitating perhaps further visits to the school.

32 For the why discussion, see the see the pathways section in the conclusion. 17
with teachers, and the linkage with school principals and teachers. The training intervention showed the least significant effects, a finding consistent with the lack of impact found on learning outcomes; while the largest totals were with linkage and linkage+election interventions, suggesting that the significant effects indeed indicate a pathway towards learning. Surprisingly we find the smallest share of significant effects for the school committee, which was the focus of the pilot.

# _School committee and community support for education (Tables 10-13)_

Here we examine whether any of the interventions has an effect on school committee activities, engagement with the community and perceptions of effectiveness, by the principal, teachers and school committee representatives.<sup>33</sup> First we look at whether the school committee improves interaction and outreach, specifically meetings (e.g., with members only, principals, parents, district or sub-district education offices, village council, teachers), shown in Table 10. Naturally, the number of meetings does not indicate the _quality_ of interactions among stakeholders, but the quantity is at least the first step in improving interaction. On average, school committee meetings with most stakeholders increase, corroborating analysis from midterm data that the grant funds were primarily used for meetings.<sup>34</sup> However, only the grant intervention has a significant effect on the summary index, with individual effects demonstrated for internal meetings (increase by 0.95 meetings) and those with the local education office (increase by 0.32 meetings). Interestingly, with the linkage interventions, meetings initiated by the local education office actually declined significantly, suggesting that the local education office may have pulled back as a result of greater school engagement with a more decentralized administrative body.

While grant and to a certain extent linkage help explain school committee interactions, election interventions improve perceptions of school committee effectiveness, especially by teachers, indicating that the democratic election improves legitimacy (Table 11). The school committee effectiveness summary index is significantly positive for election, linkage+election and training+election, as it is for teachers‟ perceptions of school committee effectiveness. Training in combination with elections makes the effect stronger than the effect of just election alone. However, while perceptions by teachers improve, they are not echoed by principals and school committee representatives. Across all interventions, principals don't report any expansion of school

> 33 We were not able to use the variable on parents‟ perceptions of school committee effectiveness due to high numbers of don't know responses. We added the percent of parents being able to answer this question to the composite parents' awareness of school committee, and  saw no change from any intervention.

> 34 Midterm spending data indicate that on average 43 percent of the grant is used for meetings, which is the plurality of the spending. The second largest category is “other” at 28 percent, followed by 7 percent for extracurricular activities, and four percent for improving school buildings.

18
committee outreach or involvement, and school committee representatives are no more likely to say that the school committee helped meet schools‟ needs over the last semester.

As shown in Table 12, all linkage-related interventions enhance collaboration between the school committee and the village council, as was expected with the treatment. Elections and training in addition to linkage strengthen the effects. Principals, but not school committee representatives, demonstrate improved satisfaction with village councils‟ attention to education in the village.

Along with not seeing any change in village council attention to education in the village above, school committee representatives also perceive no changes in community support in areas such as in-kind or financial contributions (see Table 13). However, parents agree with principals‟ more positive view about community engagement. Parents‟ satisfaction with community support for schools also improves with the linkage intervention, and according to data from principals, training has a positive effect on collaboration with non-educational community organizations other than the village council.

# _Parents’ effort, engagement and support (Tables 14-16)_

Here we look at how interventions affect variables related to parents' engagement in their children‟s education and parental awareness of the school committee.

The most interesting and significant finding related to parental effort, as shown in Table 16, is the total number of minutes that household members accompany a child studying at home in the past week. With the election intervention, this increases by 80 minutes (ITT, 150 minutes for IV) per week.<sup>35</sup> Statistically weak yet positive effects on homework support are found for the training+election intervention and on the summary index for the linkage+election intervention (i.e., significant for the entire index shown in Table 16). However, parental engagement is limited to supporting homework. There is no change in stakeholder (parents themselves, school committee, teachers or principals) satisfaction with parents' support for pupils' learning, nor do we see any change in the number of times parents come to school to meet a teacher or observe class. (Actually, with the training intervention, the number of times parents come to school to observe class goes down.) It is plausible that elections could affect parental effort at home, while not (yet) affecting how parents engage with the school. If elections merely promote awareness about the importance of education, without clear instructions about what parents could do, their first instinct may be to take action where they have greatest influence, at home.

> 35 The survey asked fathers, mothers and anyone else in the household to report the number of hours per day and the number of days a week that they accompanied his/her child studying. Field workers were then instructed to convert hours per day to minutes, so there was the potential for rounding errors, and we see this with peaks in the data on the hour (e.g., 60, 120, 180 minutes).

19
Next we look at the first order effect of whether parents simply know the school committee exists, and if yes, whether parents know any members. Naturally, this indicator doesn‟t signal parental engagement, but such awareness could be a first step in school-level participation. The grant intervention increases parents‟ knowledge of the school committee by 13 percentage points and improves parents‟ ability to name members of the school committee (see Table 15). This could be due to the grant attracting community-wide attention, increased outreach by school committee members or a greater number of parents attending school committee meetings. Interestingly, while awareness of the school committee improves, as shown in Table 15, we see no change in the proportion of parents being able to answer a suite of questions about the school committee (such as whether the school committee fights for parents interests or seeks input from parents and the school, and whether the committee chair can influence principals‟ decision-making on school policies). Thus, while basic awareness improves, this does not extend to knowing what the school committee does or believing that it advocates for parents' needs. Also, although awareness improves, as shown in Table 14, school committee representatives indicate no effect on parents‟ in-kind or financial support for school committees, consistent with no change in community support in this area (see Table 13).

# _School-based management and accountability (Tables 17-19)_

In this section we look at how interventions affected variables related to school based management: principals‟ accountability and effort, and teacher management.<sup>36</sup> An objective of the interventions was to promote accountability for the schools‟ routine financial decisions, and we find that the grant has a positive effect on the accountability of principals to parents and the school committee (summary index as a whole is significant, see Table 18). Principals report to provide more information to parents about school funding and budgeting (significant for the linkage interventions); although this is not confirmed by parental reports, which show no change. We see no change in other accountability measures, such as principals sharing the budget with the school committee, or principals involving the school committee and community in developing the school budget.

In the area of teacher management, as shown in Table 19, we that we find teachers and principals respond positively to the linkage+training intervention. Teachers‟ overall assessment of principals improves by four percentage points, possibly because principals report they are more likely to reward teachers who perform

> 36 More or different kinds of teachers, such as contract teachers, could also lead to learning; but we find that the absolute number of teachers does not change significantly as a result of the interventions (see Table 17). (Of course, the school could have dismissed teachers and hired new ones, but we do not have data on this.)  We think this result is due to the likely already sufficient number of teachers in the schools (on average nine civil servants and contract teachers at baseline) and against the backdrop of a country with an oversupply of teachers (World Bank, 2010). 20
well through recognition, gifts or money, which also improves with linkage alone. However, teachers report a _decline_ in rewards and sanctions by principals with the linkage intervention, by around 2 percentage points.

Regarding interactions between teachers and principals, we find no change in the number of teacher evaluations conducted by principals across all interventions. Teachers report that elections and linkage+elections result in more routine meetings between principals and teachers, although this is not substantiated by principals.

# _Teacher motivation and effort (Table 20)_

In this section we examine perceptions of quality and performance of teachers (by the principal, school committee, parents and teachers themselves), effort demonstrated by teachers (measured by hours spent on teaching activities) and direct classroom observation of teaching (if there were students in the classroom, was there also a teacher).<sup>37</sup> The results are presented in Table 20.<sup>38</sup> We see no effect on respondents‟ perceptions of teacher effort. However, we see changes in the two more objective variables -- observing teachers in the classroom, and teachers putting in more hours per day.

To measure teaching hours, we look at self-reported hours per day that teachers spent on lesson preparation and grading (in school and out of school), and teaching and supervising class activities (in school only).<sup>39</sup> These hours do not include training, administrative work or giving assignments without overseeing students doing them. With the election intervention alone, teaching hours per day increases by 0.63 hours (ITT; 1.2 hours IV), and by 1.1 hours (ITT; 2 hours IV) for the linkage+election intervention. This suggests that teachers respond to the linkage and election interventions with increased effort, especially election (perhaps community representation improved legitimacy), and linkage+election (oversight and support from a local administrative body may further increase legitimacy, and bring attention to the importance of education). It's possible that the election and linkage+election interventions also provide teachers with more influence and voice, since traditionally school committee representatives were appointed by principals. These interventions shift the power of principals, and increase the profile of education in the village, which may contribute to teachers' job satisfaction and overall community respect for teaching.

While teaching hours increases with election and linkage+election interventions, a puzzling result is that the proportion of teachers present in the classroom at the time of the survey (when there were children in the

> 37 For this variable, we divide the number of classrooms with teachers by the number of classrooms that fieldworkers observed with students in them.

> 38 We did not measure teacher absence since this required a team of enumerators to make surprise visits to the school, which was not supported by the limited budget for the evaluation.

> 39 The question asked of teachers was hours per week in the past week they engaged in a particular teaching activity, and this was divided by number of days per week that the teacher worked in the past week. 21
classroom) _decreases_ with election (by 5 percentage points ITT, 9 percentage points IV) and training+election interventions (by 7 percent point ITT, 12 IV). So while elections may prompt teachers to increase working hours, there is a significant decrease in the fraction of teachers in the classroom. Particularly for elections, and to a lesser extent for training +elections, these results seem to contradict each other. For this reason we also analyze the impact on each of sub-components of the hours worked variable (not shown in a table). The results indicate that for election it is the lesson preparation time that went up and for training+elections the time grading. These are both non-teaching activities, making it less surprising that no positive effect was found for the fraction of classes observed with a teacher present. For the linkage+election intervention, where there is no contradiction, teaching and grading hours go up, confirming that the intervention has an effect on the time teachers teach children in class.

# **VI. Comparison with qualitative study**

In addition to variables described above and analyzed as part of our quantitative work, we also collected indepth information through focus groups and community interviews about how school committees and communities were responding to the interventions, in a sub-sample of six, purposively-selected schools. Field teams spent about a week with school committees and other education stakeholders in July 2008, just several months before the endline survey (see Table 2) **.** Results in detail are presented in Bjork (2009); in this section we summarize the main conclusions, and compare findings with those from the quantitative research.

The primary focus of the qualitative work was to look at a combination of interventions and their synergies, so the research team selected schools exposed to the linkage+training, linkage+election, training+election treatments. For each combination, the project implementer was asked to select a school where the school committee appeared to be functioning well, and another where it was operating with difficulty. The qualitative and quantitative research were driven by similar evaluation questions, but the qualitative research concentrated on the influence that school committees exerted on school-based management variables, such as parental participation, decision-making, and the quality of services provided in the schools, rather than learning.

The qualitative research underscores school committees‟ appreciation for receiving funding that was directly under its control. Respondents indicated that the grant was the impetus for more face-to-face dialogue among members, which is in line with midline quantitative findings, that the grant is mostly used for meetings. While the grant fostered dialogue, it also caused conflict between the school committee and principal in some cases. The fact that the principal and school claimed authority over the grant indicates that the grant was successful in serving as a counterbalance to principal influence. One other change we see from the 22
quantitative analysis is improvement in parent awareness of school committee existence or its members, yet the qualitative analysis shows no indication of this change.

Challenges in implementing elections, specifically that existing school committees and school management sometimes resisted membership changes, are also seen in the qualitative work, which cites one example where the election was staged and controlled by the existing school committee and school management. Not surprisingly, in the school with the staged election, respondents reported no benefits. However, where the election was conducted as designed, elections served to enhance community‟s awareness about the school committee, attract a broader spectrum of the community to serve on the school committee, and play a role in developing school committee legitimacy. This new legitimacy is also confirmed by teachers in the quantitative analysis, which shows that teachers‟ opinions about school committee effectiveness improve (for election alone and combined with training). In the quantitative analysis, parents report improved satisfaction with support from the community with elections, which is consistent with qualitative findings that the advent of elections causes committee members to report that the committee represents community needs rather than those of the school administration.

Overall, consistent across qualitative and quantitative findings, we see little improvements as a result of training or training-related interventions. Trainees reported that they appreciated the opportunity to interact with fellow school committees and visit a model school committee, but they felt that the training materials lacked relevance and practical application. In the quantitative research we find that training improves school‟s cooperation with non-educational community organizations (as reported by the principal), so indeed the training may have improved knowledge-sharing and facilitated greater outreach. There is some indication that elected school committees benefit more from the training, as they were more motivated to come up with new actions, and this is at least substantiated by teacher perceptions, as their opinion of school committee effectiveness improves with training+election.

Compatible with quantitative findings, the qualitative research shows little positive effects of the linkage intervention on school-based management variables. The qualitative study finds the most common observable effects of linkage are village council members being elected or appointed to serve on the school committee; and that linkage serves existing power structures. The partnership between the school committee and village council often results in concrete actions, such as enforcing study hours in the village, which didn‟t necessarily involve actions by the school committee.<sup>40</sup> The quantitative research confirms the lack of impact of this intervention on school governance variables (such as changes in principal behavior or

> 40 Study hours were quiet time during which television and computer gaming is forbidden.  Other actions include collaborative work in building school facilities,  hiring a contract teacher, making land available for infrastructure expansion, resolving a conflict between two schools in a community, and encouraging social and religious activities at school. In some cases, village council representatives became members of school committees. 23
accountability), and that the intervention has no effect on opinions of school committee effectiveness or awareness of the school committee. The quantitative research however shows by far the biggest impacts with the linkage interventions, with particular response from teachers (with linkage+election), and learning, which was outside of the scope of the qualitative research. Although this intervention was designed to enhance the role of the school committee, quantitative and qualitative analysis are consistent that it does not achieve this. Instead of working through the school committee, the linkage intervention led to actions that bypassed the school committee, yet it was nevertheless effective in improving learning.

# **VII. Pathways to improved learning: Why did learning improve for linkage and linkage+election interventions?**

Since we find significant effects in learning only with the linkage and linkage+election interventions, here we discuss what factors may have led to these improvements. As mentioned above, we see linkage alone resulting in 0.17 standard deviation change in test scores for Indonesian, and linkage+election improving Indonesian test scores by 0.22 standard deviations (ITT/0.38 for IV).

While the linkage evaluation shows impressive results in learning and cost benefit, we don‟t find strong evidence that school-based management was an intermediate step between the intervention and improved test scores. Not surprisingly, the most obvious impact of the linkage-related interventions was school committee representatives and school principals reporting greater collaboration between the school and the village council. This collaboration was generally in the form of financial or in-kind assistance, other miscellaneous assistance such as services or administration, procurement, or manpower. While we have evidence that some kind of cooperation between the school committee and village council happened (see Table 12), this does not seem to have resulted in enhanced stature or empowerment of the school committee as a whole. Neither principals nor teachers show a change in their assessment of school committee effectiveness, and even school committee representatives themselves report no change in their opinions about whether the school committee helped meet school needs. While parents report increased satisfaction with support from the community for schools (which may not mean school committee), they don't show any change in knowing that the school committee exists or knowing its members; and parents are not in any way more involved in their child's education. We find the most compelling pathways explanation, drawing from the qualitative work, to be that linkage generated interest by elites in the community who promoted concrete actions, such as enforcing study hours in the village, but this didn‟t necessarily involve the school committee. Thus, linkage has an effect from bypassing the school committee, rather than working through it.

Another possible contributor to the success of linkage was that the intervention resulted in community-level stakeholders being assigned a specific task or agenda. Drawing on international evidence, Nguyen and 24
Lassibille (2008), Björkman and Svensson (2009), and Banerjee et al. (2010) show examples of community groups responsible for following up on an action plan for school improvement, monitoring compliance with an action plan for health service improvement, monitoring contract teacher performance, or teaching children to read. The collaborative task in the linkage intervention was to, among the school committee and village council, decide how the block grant should be utilized. This task appears to motivate community support, as in other countries.

The pathways to learning with the linkage+election intervention are clearer -- through actions of the school committee and teachers. As with linkage alone, we see parents demonstrating increased satisfaction with support from the community for the school. A higher share of parents also at least know the school committee exists (from about 50% at baseline, improved by 7 percentage points ITT/15 percentage points IV, see Table 15), although parents are not more likely to be able to name members of the school committee (improvement significant only at the 10% level). But overall awareness of the school committee is clearly improving at the household level.

At school, for linkage+election, the summary index relating to opinions about school committee effectiveness (as perceived by the principal, school committee and teachers) shows improvement (21 percentage points ITT/38 percentage points IV, see Table 11), with particular improvement around teacher perceptions of effectiveness. Teacher effort also improves, with teachers spending on average 0.6 to 1 (ITT/1.2 to 2 for IV, see Table 20) hours more per day on lesson preparation, classroom teaching and grading, inside and outside of the classroom. Teachers report an increase in meetings between teachers and principals over the past year, although this is not substantiated by principals. Thus, we hypothesize that teachers‟ effort leads to an increase in learning; and perhaps effort improves because teachers are more supported and encouraged by a school committee that holds greater legitimacy (because its members were elected by the community rather than appointed by the principal), and because it has links with a prominent, local governing body.

# **VIII. Conclusion**

In an effort to improve education quality, Indonesia has taken steps to strengthen local school governance structures and promote community engagement. The challenge in Indonesia however was that while legislation supporting school-based management signaled progress, in practice, schools and communities were not yet implementing what was stipulated in a government decree. Thus, this pilot study was designed to provide tools allowing schools and communities to implement the decree as envisioned. The tools or interventions tested as part of the pilot were a block grant to school committees, school committee training, a 25
democratic election of school committees, and facilitated meetings to promote collaboration between the committee and the village council (called linkage).

Two years after implementation, we find no effects on learning for interventions that attempted to internally strengthen the school committee – grant and training. Yet we find significant effects for interventions that sought to promote ties between the school committee and outside parties – election and linkage. We see linkage alone resulting in 0.17 standard deviation change in test scores for Indonesian, and linkage+election improving Indonesian test scores by 0.22 standard deviations (ITT/0.38 for IV).

The evaluation design team hypothesized that a stronger school committee would take on a greater role in school management, promote parental and community participation, and in turn improve learning. Yet we find little evidence that a strengthened school committee was what led to learning. The grant and training interventions, which focused on fortifying the existing committee, show little promise. Grants increase parental knowledge that a school committee exists (13 percentage points) and _internal_ school committee meetings (not between the school committee and other parties); and training promotes cooperation between the school and non-educational community organizations (11 percentage points). But these improvements are minor leading indicators towards progress in school-based management or community engagement. We don‟t see more substantial examples of a reinforced school committee, such as improved community or parental (in-kind or financial) support for the school committee, or principals involving school committees in budgeting or at least informing school committees about the school budget. These Indonesia results are in contrast to similar grants and training experiments, such as Khattri et al (2010) and Gertler et al (2010) which show learning effects. Although they don‟t show learning improvements, Gertler et al (2008) and Blimpo and Evans (2010) see changes in repetition and failure, and enrollment rates, respectively – none of which change for our evaluation.

The linkage and election interventions, which were designed to reinforce the school committee by connecting with stakeholders outside the school, prove more successful. With linkage, an intervention that shows learning improvements, we find that a local representative body supports learning by eliciting community action, such as enforcing study hours in the village, although this isn't through the school committee as intended. Parents‟ satisfaction with community support for the school committee improves with the linkage interventions, but community knowledge and awareness of the school committee doesn't change, nor do school committee activities, or stakeholder opinions of school committee effectiveness. In other contexts, such as Banerjee et al (2010) in India or Nguyen and Lassibille (2008) in Madagascar, experiments related to accountability in service delivery show improved community engagement leading to learning, in spite of rather than with help from local administrators. Yet in the linkage case, it was a local governing body that proved effective, rather than the committee.

26
In contrast to linkage, the election intervention reveals a realization of hypothesized pathways from schoollevel engagement to learning, as envisioned in Figure 1. The election intervention shows gains in two key intermediate outcome variables – teaching effort (increase by 0.6 hours per day)<sup>41</sup> and time household members spend helping children with homework (on average an increase of 80 minutes per week). With the election intervention, we also see teachers‟ perceptions of school committee effectiveness improve.<sup>42</sup> While the improvements in teacher and parental effort with the election intervention are remarkable, without the linkage intervention, elections do not translate in to improved learning outcomes. Instead, a democratically chosen school committee, combined with a school that cooperates with influential leadership in the village, improves learning. With linkage+election, teaching hours increase by 1 hour per day, and we see modest improvement in parents simply knowing that the school committee exists (15 percentage points). But this is as far as parental engagement goes – parents don‟t come to school or meet teachers more often, and are not more knowledgeable about what the school committee does. This means that while it was the school committee interventions were designed to improve parental engagement with the school and school committee, parents actually only took action at home.

These results suggest that reaching out to education stakeholders outside the school committee – through a democratic election process or by collaborating with the village council – are, in the Indonesia case, the most promising paths to improved learning. Bolstering internal school committee operations have only marginal impact, and are not cost effective. Training did not show changes in learning, was least effective at improving intermediate variables, and was the most costly (after the grant). In contrast, linkage was the least expensive and also the most cost-effective intervention (0.13 standard deviation improvement per 100 USD). This experiment shows that school-based management has potential for generating education improvements in Indonesia; but it also demonstrates that providing a school committee with resources and skills alone is insufficient. Broader stakeholder participation, which may or may not involve on the school committee, is necessary for advances in learning.

# _Postscript: progress to date_

While we can‟t draw a causal link between the study findings and influence over national policy, a positive development so far is that a recent Ministry implementation document states that school committees should

> 41 Although, as noted above, we observe 5-9 percentage point difference in the number of teachers present in the classroom for election and training+election interventions (ITT).  We cannot offer an explanation for a decline in teacher presence; but we know that the increased teacher effort was not in the classroom, rather on activities such as grading.

42 Also improves with linkage+training and training+election interventions. 27
be democratically elected, using the exact election process piloted during this study (Departemen Pendidikan Nasional (2009)). The document recognizes that in practice school committees are still chosen undemocratically and suggests that the process used for this experiment may encourage broader representation and voice.

28