metadata: authors: 0: author: Cerdan-Infantes,Pedro
1: author: Filmer,Deon P.
count: World
docna: 0: docna: Information, knowledge and behavior:
            evaluating alternative methods of delivering school
            information to parents
docty: Policy Research Working Paper
subtopic: Educational Institutions & Facilities,Educational Sciences,Educational Populations,Education For All,Education for Development (superceded),Effective Schools and Teachers
teratopic: Education
abstracts: cdata!: Improving education outcomes by
            disseminating information to parents and thereby encouraging
            them to become more actively engaged in school oversight is
            attractive, since it can be done relatively cheaply. This
            study evaluates the impact of alternative approaches to
            disseminating information about a school grants program in
            Indonesia on parents' knowledge about the program in
            general, knowledge about the implementation of the program
            in their child~^!!^s school, and participation in school
            activities related to the program as well as beyond it. Not
            all dissemination approaches yielded impacts, and different
            modes of dissemination conveyed different types of
            information best, resulting in different impacts on
            behavior. Specifically, the low-intensity approaches that
            were tried—sending a letter from the principal home with the
            child, or sending a colorful pamphlet home with the
            child—had no impact on knowledge or participation. On the
            other hand, holding a facilitated meeting with a range of
            school stakeholders or sending targeted text messages to
            parents did increase knowledge and participation.
            Facilitated meetings mostly increased overall knowledge and
            fostered a feeling of transparency on the part of parents,
            which resulted in greater participation in formal channels
            for providing feedback to the school. The text messages
            increased knowledge about specific aspects of the program,
            such as the grant amount, and tended to increase
            participation through informal channels.
repnme: repnme: Information, knowledge and behavior:
            evaluating alternative methods of delivering school
            information to parents
display_title: Information, knowledge and behavior:
            evaluating alternative methods of delivering school
            information to parents
listing_relative_url: /research/2015/04/24310282/information-knowledge-behavior-evaluating-alternative-methods-delivering-school-information-parents
topicv3: Teachers Management m1326119 1523,Teacher Absenteeism m1327035 1528,Education m1327048 470,School-Based Management m1327730 2143,Education and Culture m1327747 1472
publishtoextweb_dt: 2015-04-06T00:00:00Z
dois: 10.1086/712490
historic_topic: Education
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
no_of_pages: 39
author_keys: 0: author_key: 857582
1: author_key: 97345
model_extractions: WPS7233 Policy Research Working Paper 7233 Information, Knowledge and Behavior Evaluating Alternative Methods of Delivering School Information to Parents _Pedro Cerdan-Infantes Deon Filmer_ Development Research Group Human Development and Public Services Team & Education Global Practice Group April 2015
Policy Research Working Paper 7233

# **Abstract**

Improving education outcomes by disseminating information to parents and thereby encouraging them to become more actively engaged in school oversight is attractive, since it can be done relatively cheaply. This study evaluates the impact of alternative approaches to disseminating information about a school grants program in Indonesia on parents’ knowledge about the program in general, knowledge about the implementation of the program in their child’s school, and participation in school activities related to the program as well as beyond it. Not all dissemination approaches yielded impacts, and different modes of dissemination conveyed different types of information best, resulting in different impacts on behavior. Specifically, the low-intensity approaches that were tried—sending a letter from the principal home with the child, or sending a colorful pamphlet home with the child—had no impact on knowledge or participation. On the other hand, holding a facilitated meeting with a range of school stakeholders or sending targeted text messages to parents did increase knowledge and participation. Facilitated meetings mostly increased overall knowledge and fostered a feeling of transparency on the part of parents, which resulted in greater participation in formal channels for providing feedback to the school. The text messages increased knowledge about specific aspects of the program, such as the grant amount, and tended to increase participation through informal channels.

This paper is a product of the Human Development and Public Services Team, Development Research Group; and the Education Global Practice Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at pcerdaninfantes@worldbank.org and dfilmer@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Information, Knowledge and Behavior: Evaluating Alternative Methods of Delivering School Information to Parents**

Pedro Cerdan-Infantes and Deon Filmer<sup>1</sup> _The World Bank_

**JEL Classification** : I21, I28, H52, O15

**Keywords** : Economics of Education; Accountability; School Based Management; Information and Communication Technology

> 1 We thank Santoso and Mayla Lestari for excellent research assistance, as well as Dina Abu Ghaida, Nazmul Chaudhury, and Futoshi Yamauchi who provided valuable comments on an earlier version of this paper.  All errors and omissions are, of course, our own.  Funding for this work was provided by The World Bank and the Dutch Government.
# **1. Introduction and Motivation**

Many countries have embraced school-based management reforms to improve the quality of their education systems, based on the notion that decision-making will be improved by devolving responsibility to those who can more easily identify the needs of schools (Barrera-Osorio, Fasih and Patrinos 2009). Decentralization also has the potential of increasing the accountability of school-level decision-making agents to local communities, which could in turn increase effort and ultimately learning outcomes (World Bank 2003).

The evidence on the effectiveness of school-based management reforms in low and middle income countries is, however, mixed.<sup>2</sup> In Nepal (Chaudhury and Parajuli 2010) and Mexico (Gertler, Patrinos and Rubio-Codina 2012; Skoufias and Shapiro 2006) school-based management reforms led to increases in school participation and grade progression, but not to significant increases in test scores. In the Philippines, impacts were documented on participation, grade progression as well as learning outcomes (Khattri, Ling and Jha 2012; Yamauchi 2014). In Madagascar, there were no detectable impacts (Glewwe and Maiga 2011). One reason for these differences is that the details of implementation matter for success (Bruns, Filmer and Patrinos 2011). In Indonesia, alternative approaches to supporting the decentralization process led to different outcomes: complementary interventions that led to increased engagement of parents with the school were associated with positive impacts on learning; other approaches were not (Pradhan et al. 2014).

There are clearly a number of potential factors that modulate the extent to which school-based management reforms translate into better learning outcomes. Building on the earlier results from Indonesia, this study focuses on how to build parent engagement by providing information.

Disseminating information to parents in order to improve education outcomes is an attractive approach since it can be done relatively cheaply, especially if the information disseminated has already been collected as a part of routine administrative tasks (Hoxby 2002).<sup>3</sup> There are three

> 2 The evidence for high income countries is likewise mixed (Hanushek and Jorgenson 1996).

> 3 The World Development Report 2004 “Making Services Works for Poor People” (World Bank 2003), argues that lack of information weakens client power (to hold providers accountable) and it weakens citizens’ voice (relative to policy makers and politicians). Information-based interventions can affect relationship of school (provider) accountability and sooner or later education intermediate outcomes (i.e. teacher attendance, facility improvement, textbook provision) and ultimately student learning outcomes. When looking at schools as a service provider, greater

2
main ways that information has been argued to have the potential to change education—and learning—outcomes.<sup>4</sup> First, information about the returns to schooling provided to students and their parents can change their valuation of the benefits of staying in school. Interventions that disseminated such information in Chile (Dinkleman and Martinez 2013), Dominican Republic (Jensen 2010), Madagascar (Nguyen 2009) and Malawi (Dizon-Ross 2014) led to greater school participation, and improved learning outcomes in the case of Madagascar. A similar type of intervention in China, however, did not yield positive impacts (Loyalka et al. 2013). Second, information about school quality can potentially lead to competitive pressure for schools to increase their performance. Provision and dissemination of such information improved learning outcomes in private schools in Brazil (Camargo et al. 2011) and Pakistan (Andrabi, Das and Khwaja 2014), although no impact was found in Chile (Mizala and Urquiola 2011). Third, information about the learning outcomes of their children, about the quality of their children’s school, or about rights and responsibilities of stakeholders can potentially lead parents to demand better services. One channel for this is the exertion of pressure on school system managers to improve the quality of provision—where school management allows for such pressure (Bruns, Filmer and Patrinos 2011). The information interventions we study here are designed to enhance parents’ knowledge about a school grants and decentralization program in order to enable them to use that information to pressure school managers—potentially through School Committees— to use those grants to improve school outcomes.

An early influential set of studies analyzed the role of a public information campaign to increase parents’ knowledge about a school capitation grant in Uganda. The findings suggested that the campaign—which included dissemination of school-by-school information on the magnitude of grants due to each school in local newspapers, as well as the posting of that school-specific information on bulletin boards within schools—led to substantial reductions in leakage of the grants (Reinikka and Svensson 2004) and increases in school enrollments and learning outcomes (Reinikka and Svensson 2010). But attempts to systematically assess the evidence on the extent to which information interventions improve outcomes suggest that effectiveness is mixed and

> (public) information about schools’ resource such as BOS funds and schools’ performance such as student test scores would increase client (i.e. parents and students) power to select between schools and to hold schools accountable for efficient use of resources. This situation would eventually trigger schools to try to improve service delivery (WDR 2004, Bruns et al. 2011).

> 4 Murnane and Ganimian (2013) usefully reviews this literature.

3
likely dependent on details of implementation as well as of context (Bruns, Filmer and Patrinos 2011; Krishnaratne et al. 2013; McEwan forthcoming; Murnane and Ganimian 2014).

For example, in three states in India, an information campaign conveying the roles and responsibilities of school oversight committees through multiple approaches resulted in significant and positive impacts on teacher attendance, the percentage of students receiving scholarships, and student reading ability (Pandey, Goyal and Sundararaman 2009; 2010). In contrast, a campaign to publicize roles and responsibilities in the Jaunpur District of Uttar Pradesh in India (Banerjee et al 2008) found impacts on knowledge about roles and responsibilities, but little impact on learning outcomes. While it is hard to assess exactly why these two studies came to such different findings, it is likely that the longer timeframe of the former—which included repeat visits to sensitize stakeholders—may have been a relevant difference.

While information provision may affect behaviors and outcomes, its effectiveness will depend on the extent to which it is understood. The capacity of the various stakeholders, and in particular parents, will therefore be important to translating that information into action (Barrera-Osorio, Fasih and Patrinos 2009). Analysis of a school-based-management reform in The Gambia highlighted the constraints that low literacy of parents placed on the absorption of information about their children’s performance and on the functioning of schools (Blimpo and Evans 2011). Whereas the reform yielded positive impacts in localities where baseline adult literacy rates were high, impacts were small (and in some specifications negative) where baseline adult literacy rates were low. Analysis of the relationship between decentralization of decision-making to the school level and learning outcomes across countries suggests that the association is positive when country-capacity is high, and can be negative when it is low (Hanushek, Link and Woessmann 2013).

Missing from much of the discussion of the impact of information provision on outcomes is the way information is conveyed to stakeholders. Studies have provided information in a number of different ways: through village or school meetings (Pandey, Goyal and Sundararaman 2009, 2010; Banerjee et al. 2010), through posters at schools (Reinikka and Svensson 2004, 2010), through “school report cards” distributed to parents (Andrabi, Das and Khwaja 2009), through 4
newspapers (Mizala and Urquiola 2011; Reinikka and Svensson 2004, 2010) or through meetings at students’ homes (Jensen 2010).

The objective of this study is to evaluate the impact of alternative approaches to disseminating information to parents on parental knowledge and participation in school management. The experiment consists of a group of four randomized control trials (RCT) implemented in three districts in Indonesia. The overall objective was to identify effective school level interventions that could be scaled up to ultimately enhance the effectiveness of school-based management of a school grant program. We focus on answering two main questions: (i) did the provision of information actually lead to increased knowledge of the program and, in turn, increased participation in school management? and (ii) which methods were most effective at increasing knowledge and participation?

The analysis yields two main findings. First, the provision of information can lead to increases in knowledge and participation in school management, but not all methods do so. In this experiment, the provision of information through a facilitated school meeting and through SMSs (text messages) proved effective, but the provision of written materials to parents through their children did not show any impact on knowledge or participation. Second, the findings suggest that the effectiveness of alternative methods of information provision differs according to the type of information provided and the objectives of this information provision. The specificity of the information, as well as the channels of parental participation (for example, through increased parental presence at formal events at the school versus through informal communication channels) are two factors that seem to affect the effectiveness of these interventions. Facilitated school meetings increased knowledge and parent perceptions of school openness and transparency; SMSs were more effective at communicating specific information and led to more informal engagement on the part of parents.

The paper is structured as follows. Section 2 presents a short overview of the program, the interventions aimed at disseminating information, and the design of the evaluation of those interventions. Section 3 presents results (including validation of the randomization design). Section 4 concludes with policy implications.

5
# **2. The program, interventions, and evaluation design**

# **_2.1 The BOS Program_**

Indonesia has been implementing school-based management reforms over the last decade through the School Operational Assistance programme (BOS). The program, initiated by the Ministry of Education and Culture<sup>5</sup> in 2005, provides all Primary and Junior Secondary schools (public and private) with block grants, which are meant to cover all operational costs of running a school, except salaries of staff who are civil servants. The program currently provides grants to 228,000 schools benefiting an estimated 43 million students. In 2012, Rp.23.5 trillion<sup>6</sup> or 8.1% of the total government education budget, was spent on BOS and there are plans to extend BOS to senior secondary education.

Along with the school grant, the BOS program also provides schools with greater autonomy and flexibility in managing their resources and in turn, schools are expected to use these resources in a transparent and accountable way. The program aimed to strengthen school planning and budgeting processes, as well as to strengthen parental and community oversight. In particular, School Committees were given formal roles in developing school development plans and in programming the BOS resources. These reforms, coupled with the decentralization of the responsibility of the oversight of basic education from the central government to district governments, have fundamentally transformed how the education system is managed in Indonesia.<sup>7</sup> Despite these efforts, parental participation in school decision making was still low at the time of this experiment. In a nationally representative survey on school-based management carried out in 2010-11 (World Bank 2012), two-thirds of schools reported making decisions without the participation of the School Committee or parents. Most schools reported making decisions “by consensus,” usually led by the principal. In 22 percent of schools, principals alone reported making all decisions. School Committees were largely ineffective at playing the mediator role

> 5 Then called Ministry of National Education (MoNE)

> 6 The exchange rate at the time was roughly IDR9400 to US$1.

> 7 School-based management is a form of decentralization in education service delivery, devolving a degree of authority from the government to the school level. Education Law 20/2003 and Minister of Education and Culture Decree 044/2002 on School Committee and District Education Council have laid the ground for Indonesia’s schoolbased management. World Bank (2012) provides a succinct account of the status of SBM implementation in Indonesia.

6
between community, parents and school that they were intended to play (World Bank 2012; Pradhan et al. 2012).

This low level of participation was partly due to inadequate parental knowledge about the program. A survey conducted by the Ministry of Education and Culture in a nationally representative sample of parents in 2009 found that, while parents were generally aware of the existence of the program, they were much less knowledgeable about the detailed workings of BOS—knowledge necessary to actively participate in school affairs and hold schools accountable for the use of BOS funds. Even general knowledge about the existence of the program was far from universal, with only about 60 percent of parents aware of its overall objective. Knowledge of more detailed information like the eligible uses of the BOS grants was even lower, with less than 30 percent being knowledgeable about such information, the reporting process and the amount of per-student allocations (World Bank 2012).

# **_2.2 The information interventions_**

In order to increase awareness of the program, in 2011 the Ministry of Education and Culture with the support of the World Bank, developed a Social Marketing and Information Campaign (SMIC) that included national, district and school-level interventions. The campaign included nationwide coverage (television and print media), district interventions (including local media, and district events) and different methods of delivering information to reach parents at the school level. The school-level interventions we evaluate here were part of this larger nationwide campaign. The main messages of the campaign were consistent across levels and types of interventions but the level of detail in the information provided was much more limited in the TV and radio advertisements than in the printed materials. Throughout, however, messages were deliberately simple: (1) What is BOS? (2) How much does the program provide per student to each type of school? (3) What can the resources be used for? And (4) how are parents expected to participate in managing the program?

We focus here on four approaches that were implemented in an experimental way in order to evaluate their relative effectiveness at reaching parents and affecting their behavior: (1) Physically giving information to parents through their children (which included two variants—a one-page letter and a colorful “pocket book”), (2) using a series of SMS messages to contact 7
parents; and (3) hosting a facilitated school meeting.<sup>8</sup> The messaging was designed to be simple and consistent across different delivery methods (conveying the main messages described above), with different levels of intensity—from low intensity passive interventions (letter, pocket book), to middle intensity (SMS), to high intensity (facilitated school meetings).

The ultimate objective of these interventions was to improve the use of BOS funds by increasing parental involvement in planning and overseeing the program at the school level. The better use of funds was, in turn, expected to improve overall education indicators, including learning outcomes. The theory of change therefore encompasses the following steps. First, information is provided to parents. If that information is assimilated and transformed into knowledge, it can lead to a change in behavior. This change includes operating through formal participatory mechanisms, like the BOS planning meeting that schools are required to hold every year;<sup>9</sup> parental involvement in the School Committee; or through informal channels such as providing direct feedback to teachers or the principal; or voluntary contributions to the school (either monetary or in-kind).

Examples of the types of changes envisioned include: School-provided scholarships or transportation subsidies to those in need to keep them in school (affecting enrollment and retention); using more resources for instructional and learning-related materials (leading to improvements in learning); using contract teachers more effectively and liberating resources to spend on other inputs; and, more subtly, greater engagement on the part of parents leading to greater accountability pressure on school-level actors leading to greater efforts—such as higher teacher attendance, for example. The greater oversight and input is expected to lead to more effort on the part of school actors, and more and better used resources, which would lead to better outcomes.

While the ultimate impact of reforms on learning outcomes are important, we focus here on the earlier steps in the results change, the assimilation of knowledge and changes in behavior. We do so in part because the one-year timeframe of the evaluation makes it unlikely that we would

> 8 An additional intervention, posting information on the school’s notice board, was also implemented in a separate district. However, it was implemented without a counterfactual and we therefore do not attempt to evaluate its impact here.

> 9 The BOS planning meeting is part of school planning meeting, required to be held at least once in a year. The meetings are supposed at least involve principal, teachers, and School Committee members

8
see impacts on outcomes and focusing on them could therefore be misleading (King and Behrman 2009). Our focus on knowledge and behavior is also, however, based on the independent interest in how to best affect those.

# **_2.3 The evaluation design_**

The interventions we evaluate were implemented in 3 districts of Indonesia: Tulungagung, Malang and Sumbawa.<sup>10</sup> These districts were selected based on having enough schools in the program monitoring survey to carry out the experiment with sufficient sample size, as well as on their willingness to participate in a randomized pilot. Implementation constraints associated with the program made it impossible to simply randomize the interventions across schools in the various districts. With district administrations in charge of implementing the program it was deemed infeasible to have multiple approaches plus a control group within the same district. The evaluation design, therefore, consists of a collection of three district-level experiments based on within-district randomization of schools to one type of treatment (with two variants in the case of Tulungagung) or a control group. While not threatening the internal validity of the evaluation results for each intervention, the fact that we are comparing different interventions implemented in different districts means that the comparison across interventions is less clean. The approach is arguably better, however, than the oft-made comparisons of different interventions implemented in different countries (see for example Kremer, Brannen and Glennerster 2013 or McEwan forthcoming). This is because the institutional setting is consistent across districts, and the social context is broadly similar.

A total of 148 Primary schools in 3 districts were part of the experiment.<sup>11</sup> Figure 1 illustrates the implementation of the evaluation. In Tulungagung, a total of 81 primary schools were available for the pilot. They were randomly divided into 3 groups: in the first group of 27 schools, a letter signed by the principal was given to students to pass on to their parents; in the second group of 27 schools students were given a colorful pocket book for the same purpose; the 27 schools in the third group did not receive any materials. Ten parents in each of these schools were surveyed at endline. In Malang, where there were only 26 primary schools at baseline, SMSs were

> 10 Tulungagung and Malang are in the East Java Province, Sumbawa is in the West Nusa Tenggara Province. 11 Note that the intervention was also implemented in some Junior Secondary schools.  However, because the number of those schools is small, and because Junior Secondary schools are different in a number of ways from Primary Schools, we focus only on the intervention impacts in the latter.

9
randomized at the individual parent level. Cell phone numbers and names of parents in the 26 schools were collected during a meeting at the beginning of the school year. A random sample of 20 parents in each school were selected; a target of half of them were randomly selected to receive a series of SMSs with the remainder not receiving SMSs.<sup>12</sup> All 20 parents were interviewed in the endline survey. Last, in Sumbawa, where 41 primary schools were available for the pilot, a facilitated meeting to explain the content of a poster was carried out in 21 randomly selected schools and 20 formed the control group. Twelve parents in each of these schools were randomly selected for the endline survey.

**Figure 1: Evaluation design**

<!-- Start of picture text --> Printed materials SMS School Meeting<br>(Tulungagung) (Malang) (Sumbawa)<br>81 schools 520 parents in 41 schools<br>26 schools<br>Randomly selected Randomly selected Randomly selected<br>21<br>27 27 schools<br>27 schools schools 20 schools<br>schools Pocket<br>Control School Control<br>Letter Book<br>Meeting<br>264<br>256<br>270 270 270 parents 252 240<br>parents<br>Parents Parents Parents Control Parents Parents<br>sent SMS<br>Group<br><!-- End of picture text -->

# **_2.4 Data_**

> 12 While carried out centrally, the randomization didn’t always result in equal numbers of parents selected for treatment and control in each school (22 or the 26 schools have between 8 and 12 parents in the treatment group).

10
The main source of information for this study is a survey conducted to monitor the overall effects of the campaign at the national level. This survey contains information from parents from 720 Primary and Junior Secondary schools nationwide, stratified at the province level to capture large and small provinces, in a total of 19 districts in 9 provinces.<sup>13</sup> The monitoring survey asks questions about all campaign interventions, including the TV, radio and print media advertisements. For the districts in the pilot, the survey was augmented to include questions about the school-level interventions, school-level knowledge, and participation in the school.

Even though two rounds of data were collected from each school (baseline and endline), the parents interviewed in each round were different so we do not have a panel of parents that we could follow over time. At baseline, one student from each grade (from second to sixth grade) was selected randomly, and the survey team visited the student’s home to interview his or her parents. If the parents were not home, the caretaker of the child was interviewed. For the endline survey, the number of parents in each school was increased, and varied by district in pilot schools. In Tulungagung (letter and pocket book) a total of 10 parents were interviewed in each school (2 from each grade, from second to sixth grade); in Malang (SMS), 20 parents in each school were interviewed. The parents were intended to come from all grades in school; and in Sumbawa 12 parents per school (two in each grade from grades one to six) were interviewed. The total sample in each treatment and control group is shown in Figure 1.

# **_2.5 Indicators_**

We measure knowledge and participation along three dimensions: (1) General knowledge about the BOS program, (2) knowledge about how the program operates in the school that parents belong to, and (3) participation in school management (Table 1). All individual variables are binary (yes/no) and, because of their large number, we create an index for each dimension by adding them up. The results section focuses on these indices, though we also subsequently present results on individual variables. In addition, the results section uses mainly the endline survey, because it includes more variables and a larger sample of parents. We use the baseline survey mainly to test for balance between treatment and control groups.

> 13 The survey was stratified to capture both large and small provinces, and both urban and rural districts. Importantly for our purposes, within districts, schools were selected randomly.

11
The variables under the _knowledge about BOS_ are ordered by difficulty, or specificity of the knowledge. For example, the first question asks whether the respondent has ever heard of BOS, the second question asks whether the respondent knows what BOS stands for (“School Operational Assistance”) and the third question asks the amount of the grant. In the endline survey, a question about the uses of funds was added. In addition, in the endline, we can assess whether the responses (on the amount and uses) were correct.

**Table 1: Availability of variables used to measure knowledge** **<u>and participation at baseline and endline</u>**

|Variable|Baseline|Endline|
|---|---|---|
|**_Knowledge index BOS_**|X|X|
|Heard of BOS|X|X|
|Knows what BOS stands for|X|X|
|Knows amount|X|X|
|Gets amount right||X|
|Knows negative list of uses||X|
|**_Knowledge index BOS in school_**||X|
|Knows of BOS plan document||X|
|Knows BOS reporting documents||X|
|Knows BOS use in school||X|
|Sees BOS use in notice board|X|X|
|Knows School Committee role||X|
|**_Participation Index_**|X|X|
|Visits at least 3 times a year|X|X|
|Thinks BOS use is transparent|X|X|
|Involved in BOS plan||X|
|Communicates with School Committee|or principal|X|
|Contribution (cash)||X|
|Contribution (in kind)||X|
|Note: The composition of the indices is dif<br> <br>Contribution(labor)|ferent at baseline a<br>|nd endline,<br>X|

Note: The composition of the indices is different at baseline and endline, using the available variables indexed below A second set of questions asks about the _knowledge of BOS in own school_ . This is an important distinction to make: While disseminating information about the program as a whole may have inherent benefits in terms of increased awareness of, or interest in, education, and public support for the program, this information is unlikely to have an impact on participation and schooling outcomes of interest unless it is translated into knowledge about the program at the school level. This set of questions includes whether the respondent knows the roles of the School Committee, 12
the formal planning and reporting mechanisms under the program, how the BOS funds are spent, and whether they have seen the use of BOS funds posted on their school’s notice board (schools are required to post this information in a public notice board for all parents to see). Last, _participation_ is measured through a combination of questions on the number of visits parents have made to the school, knowledge of, and involvement in, the BOS planning process at the school, having communications with the principal, School Committee or teachers, or having provided contributions (in cash, in kind, or in labor).

**Figure 2: Knowledge and participation of parents in the national sample and pilot districts, reported at baseline**

<!-- Start of picture text --> % of respondents who answered yes<br>88%<br>Heard about BOS**<br>79%<br>Knows what BOS stands 60%<br>for** 51%<br>20%<br>Knows amount**<br>12%<br>Sees BOS in notice 11%<br>Pilot Districts<br>board** 6%<br>Visits at least 3 times per 6% Non-Pilot<br>year** 8% Districts<br>Provided feedback to 10%<br>school* 8%<br><!-- End of picture text -->

- *(**) indicates that the difference between pilot and non-pilot districts is significant at the 10(5) percent level.

There are three noteworthy features of knowledge and participation at baseline (Figure 2). First, general knowledge about the program was low, both in pilot and in non-pilot districts. In 2009, four years after the inception of BOS, a surprising 19 percent of respondents with children in school did not know about the program nationwide, only slightly more than half knew what BOS stood for and almost 85 percent did not know the amount. Second, knowledge about BOS in the school and parental participation in schools were extremely low at baseline. Only 7 percent had seen the BOS information on the school boards, which is low considering that the requirement that schools post information on the school notice board was already in place at the time. Less than 10 percent of respondents had provided any feedback to the school, and only 8 had percent visited the schools more than 3 times in the past year.

13
Last, both knowledge and participation in pilot districts was higher than in non-pilot districts at baseline. The differences are all statistically significant, indicating that pilot districts are not necessarily a representative sample of districts in the country. However, the magnitude of the differences is not large enough to dismiss external validity of these results for the rest of the country. Moreover, while knowledge and participation were higher in pilot districts, they were still low in absolute terms.

# **3. Results**

# **_3.1 Balance at baseline_**

Before turning to the impact of the interventions, the first set of results we discuss are those that test the validity of the evaluation design. Because of the differences in the baseline and endline, described in the previous section, we perform two sets of tests. First, we compare summary statistics for available outcome variables at baseline for the treatment and control groups (Table 2). Second, we use the richer endline survey to compare summary statistics of fixed background variables of respondents in the treatment and control groups (Table 3).

The first set of balancing tests (Table 2) reveal no statistical differences between treatment and control groups in Tulungagung and Sumbawa in any of the baseline variables. However, in some cases, the magnitude of some differences is large with corresponding large standard errors. In part this is due to the relatively small number of observations in the baseline survey. A joint test for the joint significance of the variables (or indices) in predicting treatment fails to reject that they are jointly equal to zero, supporting the notion of baseline balance in these outcome variables.

# **Table 2: Summary statistics of available knowledge and participation variables at baseline**

14
||||**Tulungaggung**||**Malang**<br>|**Sum**|**bawa**|
|---|---|---|---|---|---|---|---|
||**Full Sample**|Treatment 1<br>(Letter)|Treatment 2<br>(Pocket book)|Control|All Schools<br>(SMS)|Treatment<br>(meeting)|Control|
|**Knowledge Index (general)**|1.46|1.72|1.58|1.55|2.21|1.59|1.50|
||(0.93)|(0.85)|(0.82)|(0.88)|(0.73)|(0.90)|(0.88)|
|Heard about BOS|0.80|0.88|0.90|0.84|0.98|0.85|0.87|
||(0.40)|(0.32)|(0.30)|(0.37)|(0.13)|(0.36)|(0.34)|
|Knows what BOS stands for|0.53|0.65|0.56|0.60|0.81|0.52|0.46|
||(0.50)|(0.48)|(0.50)|(0.49)|(0.39)|(0.50)|(0.50)|
|Knows BOS amount|0.14|0.19|0.12|0.11|0.42|0.22|0.17|
||(0.34)|(0.39)|(0.33)|(0.32)|(0.50)|(0.41)|(0.38)|
|Seen BOS use in notice board|0.07|0.04|0.06|0.02|0.31|0.09|0.10|
||(0.25)|(0.21)|(0.24)|(0.15)|(0.47)|(0.29)|(0.30)|
|**Participation Index (sum of 2**|0.16|0.07|0.13|0.08|0.37|0.16|0.19|
|**variables below)**|(0.40)|(0.28)|(0.36)|(0.27)|(0.50)|(0.37)|(0.42)|
|Visits at least 3 times per year|0.08|0.02|0.02|0.02|0.24|0.02|0.03|
||(0.27)|(0.15)|(0.15)|(0.13)|(0.43)|(0.14)|(0.17)|
|Provides feedback to principal,|0.09|0.04|0.11|0.06|0.13|0.14|0.16|
|teacher or SC|(0.28)|(0.21)|(0.31)|(0.25)|(0.34)|(0.35)|(0.37)|
|Number of observations|3,326|135|132|125|121|102|100|
|Tests of joint significance in predic|ting treatment|||||||
|Individual variables:||||||||
|F-statistic||0.38|1.33|||0.48||
|P-value||0.89|0.26|||0.82||
|**Indices:**||||||||
|F-statistic||0.55|1.26|||0.22||
|P-value||0.58|0.29|||0.80||

Note: Standard errors reported in parenthesis clustered at the school level. * used for significance levels, but none of the differences between treatment and control groups are statistically significantly different from zero at 10%. The test for joint significance consists of a joint test that all the individual variables, or the two indices, are significantly different from zero in a regression of treatment on these variables.

The additional set of balancing tests using the endline survey (Table 3) largely confirm the results of the first set for the pocket book and the school meeting interventions, where no statistically significant differences are observed in any of the background variables. In Tulungagung, however, the treatment group selected to receive the letter is more likely to live in urban areas than the control group, which could indicate higher socioeconomic status; but they are also less likely to be employed for wages, thus more likely to be self-employed, and less likely to have the parent of the child as the head of the household, variables generally correlated with lower socioeconomic status. There are no statistically significant differences in household per capita expenditure. A joint test for significance of these variables in predicting treatment rejects that they are jointly equal to zero. However, the results do not seem to indicate a systematic pattern; while some differences are statistically significant, they do not clearly signal systematic bias between the two groups. In the case of Malang, the differences are less robust: 15
there are statistically significant differences between the treated and control parents at the 10 percent level for two variables: education and whether the household head works for a wage. While these differences are statistically significant, they are generally quite small (they are on the order of 15 percent). A joint test for significance of the full set of variables in predicting treatment fails to reject that they jointly equal zero. We nevertheless control for demographic characteristics in the analysis to account for any potential differences between the different groups.

**Table 3: Summary statistics of background characteristics at end line, by treatment and control groups**

||||<br>**Tulungaggung**||<br>**Mal**|**ang**|<br>**Sumb**|**awa**|
|---|---|---|---|---|---|---|---|---|
||**Full Sample**|Treatment 1<br>(Letter)|Treatment 2<br>(Pocket book)|Control|Treatment<br>(SMS)|Control|Treatment<br>(meeting)|Control|
|Household in urban area|0.42|0.55**|0.31|0.30|0.96|0.96|0.19|0.10|
||(0.49)|(0.50)|(0.46)|(0.46)|(0.19)|(0.19)|(0.39)|(0.30)|
|HH head with junior|0.36|0.29|0.29|0.33|0.63*|0.54|0.23|0.20|
|secondary or more|(0.48)|(0.45)|(0.45)|(0.47)|(0.48)|(0.50)|(0.42)|(0.40)|
|HH head senior|0.26|0.19|0.19|0.22|0.51|0.46|0.15|0.13|
|secondary or more|(0.44)|(0.40)|(0.39)|(0.42)|(0.50)|(0.50)|(0.35)|(0.34)|
|HH is male|0.60|0.64|0.68|0.69|0.74|0.68|0.47|0.46|
||(0.49)|(0.48)|(0.47)|(0.47)|(0.44)|(0.47)|(0.50)|(0.50)|
|HH head is a parent|0.55|0.57**|0.61|0.64|0.70|0.64|0.46|0.44|
||(0.50)|(0.50)|(0.49)|(0.48)|(0.46)|(0.48)|(0.50)|(0.50)|
|HH is civil servant|0.05|0.04|0.03|0.03|0.11|0.11|0.03|0.03|
||(0.23)|(0.20)|(0.18)|(0.16)|(0.31)|(0.32)|(0.18)|(0.16)|
|HH is waged employee|0.25|0.25**|0.36|0.38|0.44*|0.38|0.19|0.16|
||(0.43)|(0.43)|(0.48)|(0.49)|(0.50)|(0.49)|(0.40)|(0.37)|
|HH monthly per capita|12.41|12.47|12.40|12.37|12.79|12.75|12.24|12.13|
|expenditure (log)|(0.63)|(0.45)|(0.46)|(0.45)|(0.55)|(0.58)|(0.54)|(0.57)|
|Number of observations|4482|270|270|270|256|264|252|240|
|Tests of joint significancein|predicting trea|tment|||||||
|Individual variables:|||||||||
|F-statistic||3.32|0.67||1.01||0.83||
|P-value||0.01**|0.72||0.45||0.58||

Note: Standard errors reported in parenthesis clustered at the school level. * (**) indicates statistical significance as compared to the control group at the 10 (5) percent level.

# **_3.2 Outcomes_**

This section organizes the results around the logic discussed above. First, we present the impact of the different approaches on whether information was assimilated and translated into increased knowledge about the program. Second, we test whether this has translated into more knowledge about the BOS program in the specific school, particularly knowledge about the formal mechanisms for participation in school and about the use of BOS funds. Last, we present the results on actual participation in the school, both through formal as well as informal channels.

16
# **_Impact on general knowledge about the BOS program_**

The first test of whether the school level interventions had an impact on any variables of interest is whether the information provided to parents translated into actual knowledge about the content of the materials. Figure 3 presents the summary results for the four approaches tested. The figure shows the mean index for knowledge about BOS in the control group for each district (hatched rectangles), and the average treatment effect of each intervention after controlling for background characteristics of the respondents (solid rectangles).<sup>14</sup> Dark rectangles for the average treatment effect represent statistically significant results (at 5% level) whereas lighter rectangles show non-significant coefficients.<sup>15</sup>

**Figure 3: Average Treatment Effect on knowledge about the BOS program**

<!-- Start of picture text --> 3.0 ATE<br>0.2<br>2.5 Knowledge about BOS<br>(control)<br>2.0<br>1.5<br>0.6<br>2.5<br>1.0 2.0 2.0<br>0.5 1.1<br>0.0<br>Letter Pocket SMS Meeting<br>-0.5<br>Knowledge Index<br><!-- End of picture text --> Note: Hatched rectangles show the mean index of knowledge in the control group of each district, solid rectangles show the average treatment effects (with dark shading indicating significance at the 5% level).

The results show that the school meeting and SMS interventions had positive and significant effects on general knowledge about BOS, but written materials did not. The school meeting had

> 14 A full set of regression results is included in the Annex. As described in the previous section the model controls for background characteristics (education, gender and type of employment of the household head—wage employee and civil servant—as well as log per capita household expenditures).

> 15 Using the sample sizes and clustering for each intervention, as well as the intra-cluster correlation within the control groups in each district (and adjusting for the fact that we include controls in the evaluation equations), we calculate that the minimum detectable effects (MDE) with a power of 0.8 are: between .32 and .40 SD for the Tulungagung interventions (written materials), .23 for the Malang intervention (SMS), and between .38 and .40 for the Sumbawa intervention (meeting).

17
the largest impact. In schools that held a meeting to communicate information to parents, knowledge of the 5 items included in the knowledge index increased by 0.6 items, a very significant increase considering that the average value of the index in control schools was 1.1 (equivalent to an increase of 55 percent, or 0.3 standard deviations).

The SMS intervention also significantly increased knowledge of the recipients, though by a smaller amount: 0.2 items. Relative to the control group, the effects are much smaller. Since knowledge was generally higher in Malang than in other districts, the increase in 0.2 items is equivalent to an increase in 8 percent or 0.1 standard deviations. Handing out materials to students to bring to their parents, whether in letter or in colorful pocket book form, did not translate into increased knowledge (the point estimates are negative, but small and statistically insignificantly different from zero).

Unpacking the results of the impact on the knowledge index confirms the stronger impact of the school meeting. Table 4 presents the average treatment effects of the four interventions on the knowledge of each individual item included in the index. As reported in Column 4, the impact of the school meeting is remarkably strong on all items: the percentage of respondents that answering yes to having heard of BOS, knowing what BOS stands for, knowing the amount and getting it right, and knowing what BOS cannot be used for, all increased significantly. Worth highlighting is the large increase in those who report knowing the expenditure items that are not eligible under BOS. This information, which is the most detailed and complex of all the information provided in the campaign, is also crucial to enforce the accountability mechanisms for BOS. One of the key roles of the School Committee and the parents in BOS planning and monitoring is to ensure that BOS funds are spent only on eligible items. The meeting was clearly the most effective intervention at conveying this information.

The results on the impact of SMSs on individual items show smaller effects, both in magnitude and statistical significance. The SMS campaign increased knowledge about what BOS stands for and the amount of the grant, but had no impact on the knowledge of negative list of uses for BOS. Still, the combined effect on these variables results in an overall positive impact on the knowledge index for recipients of the SMSs.

18
**Table 4: Average Treatment Effect** **<u>on indicators of knowledge about the BOS program</u>**

|||**Ave**|**rage Treat**|**ment Effec**|**t**|
|---|---|---|---|---|---|
||**Letter**||**Pocket**|**SMS**|**Meeting**|
|_Knowledge index BOS_|-0.10||-0.04|**0.23 **|******<br>**0.61 ****|
|Heard of BOS|0.00||**0.04 **|*****<br>0.01|**0.22 ****|
|Knows what BOS stands for|-0.06||-0.02|**0.08 **|*****<br>**0.12 ****|
|Knows amount|0.03||-0.01|**0.06 **|*****<br>**0.07 ****|
|Gets amount right|**0.04 **|*****|0.01|**0.04 **|*****<br>**0.02 ****|
|Knows negative list of uses|**-0.11 **|*****|-0.06|0.04|**0.18 ****|
|<br>Number of observations|<br>540||<br>540|<br>520|492|

Note: * (**) indicates statistical significance at the 10% (5%) level.

Consistent with the impacts on the overall index of knowledge about BOS, the impacts of printed materials are small and typically statistically insignificant.

This first set of results suggests that providing information to parents in a way that leads to knowledge and understanding is not as straightforward as one might imagine. The two lowest intensity interventions which involved simply handing out printed materials to children did not result in any increase in knowledge about the program, whereas the higher intensity interventions did.

The results presented above are intent-to-treat estimates that measure outcome changes in the group that was supposed to be treated. However, the lack of significant impacts on knowledge could be partly explained by discrepancies between intention to treat and the actual treatment, if a large proportion of parents in treatment schools did not actually receive the intervention. This may happen if, for example, children do not give the written materials that they received from the school to their parents, if the SMS did not reach the intended recipient, or if attendance at the school meeting was very low. If this was the case we would not be able to statistically observe impacts in the overall population of sampled parents even if the intervention was very effective for those exposed to the treatment.<sup>16</sup> In treatment schools, 24 percent of parents who were supposed to get the letter through their children report not remembering seeing it. For the pocket book the number is slightly lower, at

> 16 As indicated above, an additional source of concern is that the study was somewhat underpowered, meaning that it might be hard to detect statistically significant impacts even if they exist.  It is comforting that the finding of limited impacts in the printed materials interventions is primarily driven by small effect sizes, rather than large standard errors.

19
20 percent. Some leakage of these materials is expected, as children lose them on the way home or they get misplaced in the home before parents have a chance to look at them. In addition, parents may forget that they saw the letter or pocket book if the materials did not raise their interest. So the fact that 20 to 24 percent of respondents do not remember seeing the materials is somewhat expected. Moreover, it is unlikely that non-exposure in treatment schools is the reason why we do not observe impacts of written materials: exposure for the school meetings was similar, with 23 percent of respondents in treatment schools not reporting having attended the meeting, and we nevertheless observe very strong impacts for the school meeting.

In the case of SMSs, the gap was much larger: only 32 percent of those who were sent SMSs report remembering getting the messages, a very low number. Unfortunately, we do not have more information on the reasons for this mismatch: technical glitches may have prevented the SMSs from reaching their destination; the SMSs may have been sent to the wrong number (for example if parents changed their number since giving it to the school); the SMSs may have been sent to a different person from the one who responded to the survey; or the SMSs may have been simply deleted without being read if parents were unsure of its sender.

In order to assess the extent to which the intent-to-treat estimates might understate the potential for these interventions to achieve impacts, we re-estimate the models to determine the “treatment-on-the-treated” estimate (Figure 4).<sup>17</sup> Given the relatively small difference between actual and intended treatment, the treatment-on-the-treated results for the printed materials and meetings interventions are similar to the intent-to-treat estimates. The impacts for the SMS intervention are substantially larger (0.6 items for the treatment-on-the-treated model versus 0.2 items for the intent-to-treat model) suggesting that when parents remember receiving the SMS, they know substantially more about the program. The pattern of statistical significance of the results remains the same as that for the intent-to-treat estimates (insignificant for the printed materials, significant for the SMS and meetings). While these results suggest that an SMS intervention that reached its targets more consistently might have generated an overall greater impact on knowledge, it is unclear that such a program could have been easily implemented. As discussed above, there are a number of reasons why actual and intended treatment might differ,

> 17 We implement this by estimating an instrumental variables model where actual treatment (remembering receiving the letter, remembering receiving the pocket book, remembering receiving the SMS, and reporting having attended the school meeting) are instrumented with being in the treatment group for each district.

20
and these may not be amenable to simple fixes. In the discussion that follows, we focus on the intent-to-treat results because they capture the impacts of the interventions _as implemented_ which is arguably the most policy relevant measure of effectiveness. The fact that a better implemented SMS intervention might have larger impacts should nevertheless be kept in mind.

**Figure 4: Treatment-on-the-treated estimates of impact**

<!-- Start of picture text --> 1<br>0.8<br>0.6<br>0.4<br>0.2<br>0<br>-0.2<br>Tulunagung: Tulunagung: Malang: Gets Sumbawa:<br>Receives pocket Receives letter SMS Attends school<br>book meeting<br>Impact on Knowledge of BOS index<br><!-- End of picture text --> Note: Dark shading indicates significance at the 5% level.

# **_Impact on knowledge about the BOS program in the school_**

Similarly to the impact on general knowledge, SMSs and school meetings were the most effective methods to increase knowledge about the BOS program at the school. As shown in Figure 5, the magnitudes of the impact on the school-level knowledge index are similar for the SMS and meeting interventions. The SMS intervention increased the knowledge index by 0.35 items (the index consists of 5 items), and school meetings increased the index by 0.4 items. Relative to the control group, this impact for the meeting is large and corresponds to roughly doubling the knowledge index.<sup>18</sup> Not surprisingly considering the lack of impact on general knowledge about the program, the letter and pocket books also do not show any impact on school specific knowledge about BOS.

> 18 These correspond to increases of 0.12 and 0.17 standard deviations for the SMS and meeting interventions respectively.

21
Unpacking the results reveals variability across the different types of items that are affected by the SMS and school meetings (Table 5). The school specific knowledge generated by the school meeting seems to come largely from increased presence in the school: almost half of the effect comes from seeing the BOS information on the notice board and knowing how BOS can be used in the school. In addition, school meetings also have a significant effect on knowing the role of the School Committee, presumably because the School Committee was present at the meeting. The impact on the familiarity with formal documents however is smaller and less robust. On the other hand, the impact of the SMS intervention is concentrated in the knowledge of the planning and reporting documents from BOS. Unsurprisingly, considering the results on general knowledge, letter and pocket book have no impact on any of the knowledge items at the school level.

**Figure 5: Average Treatment Effect on knowledge about the BOS program in the school**

<!-- Start of picture text --> 2.5 ATE<br>2.0<br>0.4 Knowledge about<br>BOS in school<br>1.5 (control)<br>1.0<br>1.7<br>0.5 0.9 0.9 0.4<br>0.3<br>0.0<br>-0.5<br>Letter Pocket SMS Meeting<br>Knowledge Index in the School<br><!-- End of picture text --> Note: Hatched rectangles show the mean index of knowledge in the control group of each district, solid rectangles show the average treatment effects (with dark shading indicating significance at the 5% level).

22
**Table 5: Average Treatment Effect on indicators of knowledge about the BOS program in the school**

||**Av**|**erage Treatm**|**ent Effec**|**t**|
|---|---|---|---|---|
||**Letter**|**Pocket**|**SMS**|**Meeting**|
|_Knowledge index BOS in school_|0.09|-0.08|**0.35 **|******<br>**0.39 ****|
|Knows school committee role|-0.02|-0.05|0.04|**0.08 ***|
|Knows BOS plan in school|0.02|-0.01|**0.06 **|*****<br>**0.03 ****|
|Knows BOS reporting document|0.04|-0.01|**0.11 **|******<br>0.03|
|Knows BOS use in school|0.06|-0.03|**0.08**|**<br>**0.07 ***|
|Has seen BOS use in notice board|-0.01|0.02|0.06|**0.17** **|
|Number of observations|540|540|520|492|

These results suggest that the characteristics of the information provided, and the objectives of this information communication, matter for selecting the most effective method for delivering that information. While schools know that they are supposed to publicly display information about BOS on school notice boards, many of them do not. With the organization of the meeting, knowing that parents would come to the school led schools to post the information, resulting in an increase in the share of parents who report having seen the information. In addition, the meeting itself seems to have established more of a relationship with the School Committee, who was supposed to be present at the meeting. It is therefore likely that the meetings established incentives for greater transparency, and that the meetings were a means of increasing accountability of the School Committee.

The case of the SMS intervention is slightly different. While providing the same essential information as the meeting, the SMSs conveyed it in a non-interactive way, sending a series of messages containing specific information. The SMSs were clearly identified as coming from the school, the intention being that parents would believe that the message included important information. SMSs can be stored, which may facilitate the assimilation (and transfer) of the information provided. This is consistent with the patterns we observed in the general knowledge items most affected by the SMS intervention: specific information items such as the meaning of the BOS acronym or the amount of the grants. At the school level we see that SMSs increased knowledge about the details of the planning and reporting documents. Receiving these SMSs, however, did not increase the reported school-level transparency in the use of funds (for 23
example, the role of the School Committee or having seen BOS use on a notice board at the school).

Both specific knowledge (about the amounts, and about planning and reporting processes) and increased transparency in the school are, in principle, important to increase parental participation in the school. The BOS planning and reporting documents are designed to be a formal avenue to share information with, and gain inputs from, parents at the school level. If parents are familiar with those amounts and documents, they can more easily provide input to the school and hold the school accountable for the use of funds. However, this specific information and formal avenues may lead to very limited scope for action for parents if the information only flows one way (from school to parents). Being exposed to the actual use of BOS funds in the school, forcing schools to post information on notice boards, and getting parents to come to the school to interact with the School Committee, principal and teachers may also however be necessary to hold schools accountable to the broader community. Both methods of delivering information show promising results to increase knowledge and may provide different avenues for increasing participation. We next assess their impact on actual participation.

# **_Impact on participation_**

Measuring the impact of the campaign on participation in school management is less straightforward than on knowledge. Knowledge is directly linked to the provision of information, and we expect a direct relationship between the provision of information to general knowledge of the program, and to knowledge of the program at the school level. Indeed, the campaign included information about key planning and reporting documents, as well as the role of School Committees, which are included in the school-specific knowledge index. Participation is not as straightforward. While there are formal ways for parents to participate in school management (for example being part of the BOS planning process), those are not the only ways to influence school decision making. Parents may provide feedback to teachers or the principal informally, or contribute in-kind, or contribute their time to school projects. We construct a participation index that includes 7 items: The number of visits to the school; feeling that the BOS program implementation is transparent at the school; having provided feedback to the principal/School Committee/teachers; being involved in the BOS planning process; and contributing directly to 24
the school in cash, in-kind or in the form of labor. The index is an unweighted addition of all these items. We also assess impact on each of these individual measures of participation.

Consistent with the results on knowledge, the average treatment effects on the participation index show positive and statistically significant impacts from the school meeting and SMS interventions, but not from the letter or pocket book; and the size of these effects is similar for both interventions (Figure 6). Looking at impacts on individual items reveals that the impact of the school meeting on participation is concentrated on the number of school visits and the feeling of transparency. On the other hand, the impact of SMS is mostly due to increased communication and contributions from parents (Table 6).

**Figure 6: Average Treatment Effect on participation in school**

<!-- Start of picture text --> 3.5 ATE<br>3.0<br>Participation Index<br>0.3<br>(control)<br>2.5<br>2.0<br>1.5<br>2.6<br>0.4<br>1.0<br>1.5 1.5<br>0.5 1.0<br>0.0<br>-0.5<br>Letter Pocket SMS Meeting<br>Participation Index<br><!-- End of picture text --> Note: Hatched rectangles show the mean index of participation in the control group of each district, solid rectangles show the average treatment effects (with dark shading indicating significance at the 5% level).

25
**Table 6: Average Treatment Effect on indicators of participation in school**

||**Ave**|**rage Treatm**|**ent Effec**|**t**|
|---|---|---|---|---|
||**Letter**|**Pocket**|**SMS**|**Meeting**|
|_Participation Index_|0.24|-0.01|**0.28 **|******<br>**0.36 ****|
|Visits at least 3 times a year|0.05|-0.05|**0.09**|**<br>**0.15 ****|
|Thinks BOS use at the school is transparent|0.08|0.05|0.05|**0.20** <sup>**</sup>|
|Communicates with School Committee/<br>principal or teacher|0.00|-0.02|0.04|0.00|
|Involved in BOS plan|0.00|0.00|0.02|0.01|
|Contribution (cash)|0.03|-0.03|-0.04|0.06|
|Contribution (in kind)|-0.01|0.00|**0.07 **|******<br>0.00|
|Contribution (labor)|**0.08 ****|0.04|0.05|-0.05|
|Number of observations|540|540|520|492|

The increases in the knowledge about the use of BOS, the understanding of the roles of the School Committee and the observation of the BOS use on the notice board all likely contributed to a feeling of increased transparency. The percentage of parents who report coming to the school at least 3 times per year almost doubled in schools with a school meeting as compared to control schools (37 percent in treatment versus 19 percent in control). The effect seems entirely due to the meeting itself. Parents in control schools report an average of 2 visits to the school per year, while in treatment schools the average is 2.5. So the main impact of the meeting on the participation of parents in school matters is through a feeling of increased transparency and engagement at the school. However, there were no observed impacts on direct communication with the school actors (through teachers, the principal, or the School Committee) or increases in contributions from parents (in any form).

The lack of impact of school meetings on the direct forms of communication is somewhat puzzling. Almost 80 percent of parents in the treatment district attended the school meeting, and school (principal, teachers) staff were also present along with School Committee members. The meeting was facilitated and interactive, allowing for questions and answers, so the fact that parents who attended the meeting do not report higher levels of communication is unexpected. A possible explanation is that the wording of the question resulted in parents interpreting these interactions as informal interactions outside of the school meeting. In fact, the question asked about “giving suggestions to principal/School Committee/teachers as part of BOS implementation”, which may have excluded the school meeting in the minds of the respondents.

26
Nevertheless, the lack of impact on increased communication with the school, outside of the meeting, is somewhat surprising, considering the large impacts on knowledge, both general and school specific, observed for the school meeting.

In contrast, the impact of the SMS intervention is due mainly to increased school visits and to increased in kind contributions, but not on a feeling of increased transparency. The reasons behind the positive impacts on contributions may also be due to the type of knowledge SMS were effective at generating: Because SMSs result in increased knowledge about amounts and uses, parents may feel compelled to assist the school carry out tasks that are not eligible under BOS (such as repairs or construction), leading to the increases in in-kind and labor contributions. SMSs also had an impact on the number of visits to the school, though the impact is smaller than that for the meeting.

It is of note that neither the school meeting nor the SMS increased the participation of parents in the BOS planning process, a key objective of the campaign. This highlights the limitations of formal channels of communication with parents at the school level. The BOS planning process is still largely driven by school principals, and while schools are supposed to consult parents and the School Committee is supposed to endorse the plan, in practice, very few parents report actually participating in this process. The fact that, despite large impacts on many key knowledge indicators, the school meetings and SMSs did not result in increased involvement in planning has clear implications for policies surrounding a program like BOS. In particular it suggests the need to open up the formal process to more parental scrutiny, perhaps by convening one or several meetings.

# **4. Conclusions and implications**

Before turning to the policy implications of these findings, it is important to recognize various potential shortcomings of this study. First, the exposure time was short (one year), which means that we are only able to look at the proximate determinants of impacts (that is, knowledge and behaviors) rather than on the final outcomes of interest such as student performance (Banerjee et al. 2010 showed that increased knowledge did not translate into better learning outcomes in the short term). It is also possible that longer term exposure might result in greater impacts even on the indicators of knowledge and participation. Second, the interventions we study are embedded 27
in a larger national campaign aimed at building knowledge around the program. The evaluated impacts should be understood as impacts over and above any effects that the national program had on both control and treatment groups. Third, baseline knowledge and behaviors differed across the districts in which the interventions took place. Some of the observed significant effects were found in districts where counterfactual levels of knowledge or behavior were low. If it is harder to affect change when baseline levels are high then it might not be appropriate to draw implications about relative effectiveness across districts. At the same time, some of the significant impacts we find are in settings with relatively high counterfactual levels (for example the impact of SMSs on participation) so it is not the case that we cannot identify such impacts. Last, as mentioned above, our study may be underpowered to detect impacts even when they are present. A gauge of power is the “minimum detectable effect,” which we estimate to be between 0.2 and 0.4 standard deviations depending on the outcome and district.<sup>19</sup> Our estimated impacts tend to be around (or below) those values which suggests that we may be missing true (albeit small) impacts for the printed materials interventions. However, the fact that we consistently find impacts that are close to zero across the various indicators across the two types of printed materials interventions, makes it less likely that the results are driven purely by low power.

With these caveats in mind, we nevertheless draw two main conclusions from our results. First, it is clear that the provision of information to parents does not equal knowledge and that different methods of providing information have very different impacts on the knowledge acquired by the recipients. School meetings and SMSs both had positive and significant impacts on knowledge and participation, while the provision of written materials to parents through the child (both letters and colorful pocket books in our case) did not result in increased knowledge. The two methods that proved effective in this study both deliver information _directly_ to parents, and they are both relatively “intensive” interventions. This is clear in the case of the school meeting, but it is also the case for SMSs. Our results indicate that getting a personal SMS from the school principal (even if another 20 parents got the same message) is not the same as getting a letter or pocket book through the child. It would appear that the more direct the interaction between the school and the parents, the greater the impact.

> 19 That is, the experiment would have an 80 percent change of yielding a positive effect at a 0.05 significance level if the true impact was 0.2-0.4 standard deviations or higher.

28
The second conclusion is that the nature of the knowledge and the ultimate objectives of the provision of information should matter for the selection of the method for delivering the information. School meetings were effective at generating knowledge, increasing the presence of parents in the school and improving the feeling of transparency that parents have with respect to school management. However, they do not seem to have generated increased interaction between parents and the school agents (outside of the meeting), or increased parental contributions to the school (in any form). SMSs, on the other hand, proved effective at generating knowledge about specific and detailed information (such as the amount of the grant), increasing parental awareness about the formal processes around BOS planning and reporting and, despite having no impact on participation in these formal processes, increasing participation through parental contributions.

The findings of this study have implications for the design of information campaigns and for directions for future research in this area. First, information should be delivered from school to parents as directly as possible. While traditional marketing methods (TV, radio) were not part of this evaluation, the results suggest that parents tend to pay more attention when the information is provided directly to them from the school. If the letter and pocket books did not generate any new knowledge, whether general or school specific, it is unlikely that more general marketing methods such as TV and radio will work either, though that remains to be proven and should be the subject of further study.

Second, the results point to the importance of intensive interaction between parents and the school community in the form of school meetings. The meetings showed the largest and most robust impacts on knowledge and transparency. Working closely with parents to engage them in participating in school planning is an effective way of involving them. However, this would have to be an active effort from the school, and in a context such as Indonesia’s would likely need to be mandated as part of the BOS program. As shown by the low participation and lack of impact in the participation of parents in a theoretically open process like BOS planning, just creating formal avenues for interaction without actively facilitating that process is unlikely to have much impact.

Last, the results highlight the potential that information technology has to increase knowledge and participation of parents in school management. While only SMSs were tried in this 29
experiment, other forms of social media might fall in this category. The personalization of the information at low cost, the easy storage of information and, with more recent technologies, the visually appealing delivery of information have the potential to impact the participation of parents in the management and oversight of schools. This is a relatively new area where future research has the potential to shed considerable light.

30