metadata: authors: 0: author: Asim,Salman
1: author: Riaz,Amina
count: Pakistan
docna: 0: docna: Community Engagement in Schools : Evidence
            from a Field Experiment in Pakistan
docty: Policy Research Working Paper
subtopic: Educational Sciences,Effective Schools and Teachers,Educational Institutions & Facilities,Natural Disasters
teratopic: Education,Environment
abstracts: cdata!: This paper presents the results of a
            field experiment in rural Sindh, Pakistan, where half of the
            school-age children (ages 6-10 years) are out of school. The
            study tests simple and low-intensity approaches to
            strengthen engagement of communities with schools:
            face-to-face dialogue at externally facilitated community
            meetings, and ongoing, anonymous dialogue via text messages.
            The interventions increased communities' interest in
            education as measured through an improvement in the number
            of functioning schools and, in the case of the text message
            treatment, substantial gains in retention of students in
            grades 2, 3, and 4. On the supply side, the schools
            significantly increased staffing and the share of
            one-teacher schools was reduced; however, teacher
            absenteeism increased, and there was no substantial impact
            on basic school infrastructure. Elections and capacity
            building for school committees were implemented in a
            cross-over experimental design. The intervention undermined
            the participation of communities in meetings and reduced
            impacts on all indicators except new admissions and
            availability of toilets in schools. No evidence is found of
            impact on measured test scores for any intervention.
geo_reg_key: 517198,517191,517212
repnme: repnme: Community Engagement in Schools : Evidence
            from a Field Experiment in Pakistan
display_title: Community Engagement in Schools :
            Evidence from a Field Experiment in Pakistan
listing_relative_url: /research/2020/06/32137189/community-engagement-schools-evidence-field-experiment-pakistan
topicv3: Teachers Management m1326119 1523,Teacher Absenteeism m1327035 1528,Education m1327048 470,School-Based Management m1327730 2143,Flood m1503315 3216
publishtoextweb_dt: 2020-06-16T00:00:00Z
historic_topic: Education,Environment
origu: Education AFR 1 (HAEE1)
no_of_pages: 75
author_keys: 0: author_key: 1013813
1: author_key: 1319552
model_extractions: Policy Research Working Paper 9280 Community Engagement in Schools Evidence from a Field Experiment in Pakistan _Salman Asim Amina Riaz_ Education Global Practice June 2020
Policy Research Working Paper 9280

# **Abstract**

This paper presents the results of a field experiment in rural Sindh, Pakistan, where half of the school-age children (ages 6–10 years) are out of school. The study tests simple and low-intensity approaches to strengthen engagement of communities with schools: face-to-face dialogue at externally facilitated community meetings, and ongoing, anonymous dialogue via text messages. The interventions increased communities’ interest in education as measured through an improvement in the number of functioning schools and, in the case of the text message treatment, substantial gains in retention of students in grades 2, 3, and 4. On the supply side, the schools significantly increased staffing and the share of one-teacher schools was reduced; however, teacher absenteeism increased, and there was no substantial impact on basic school infrastructure. Elections and capacity building for school committees were implemented in a cross-over experimental design. The intervention undermined the participation of communities in meetings and reduced impacts on all indicators except new admissions and availability of toilets in schools. No evidence is found of impact on measured test scores for any intervention.

This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at sasim@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Community Engagement in Schools:

Evidence from a Field Experiment in Pakistan Salman Asim<sup>¥</sup> Amina Riaz<sup>∗</sup> JEL Codes: D78, I21, I28, H75, O15 Keywords: Pakistan, Sindh, School-Based Management, Field Experiment (RCT); Information Communication Technology (ICT); Political Economy.

> ¥ Corresponding author: Salman Asim, Senior Economist, World Bank; email: sasim@worldbank.org. * Graduate student at the University of Chicago.
# Acknowledgments

We thank the Government of Sindh’s Education and Literacy Department and the Reform Support Unit, in particular, Dr. Fazlullah Pechoho, Saba Mushtaq, Faisal Ahmed Uqaili, Dr. Hamzo Khan Tagar, Muhammad Nasim Qureshi and Sania Khursheed, for extensive collaboration and assistance; the World Bank’s intervention design team, in particular Priyanka Pandey, Lorenza De Icaza and, Mariam Nusrat Adil; the Weitek Group for field implementation; Amar Fateen and Saindad Joyo for implementation support; RCons for school and household surveys; Ravinder Casley Gera for extensive support to the drafting process; Dhushyanth Raju, Nazmul Chaudhury, and Syeda Shahbano Ijaz for substantive technical discussions and edits; and Ilagno Patchamutho, Rachid Benmessaoud, Amit Dar, Keiko Miwa, Mario Cristian Aedo, Tazeen Fasih, Umbreen Arif and Margo Hoftijzer for their encouragement and support for this project. Zunaira Mughal, Robbie Dean, Joshua Gill, Cameron Friday, Ali Abbas, Amn Nasir, Tooba Akhtar, Xu Wang and Abhijit Banerjee provided outstanding research assistance at various stages of the project. The authors thank Alaka Holla, Amer Hassan, Juliea M. Trias, Deon Filmer, Owen Ozier, Achim Schmillen, Juan Baron, Koen Martijn Geven and David Evans for helpful discussions and insightful comments. We thank discussants and participants at the 2016 RISE Annual Conference at the Blavatnik School of Government, University of Oxford, 2018 PacDev Conference at University of California, Davis, 2019 AAPAM Fall Research Conference, Denver. The findings, interpretations and conclusions expressed in this paper are entirely those of the authors, and do not necessarily represent the views of the World Bank, its Executive Directors or the countries they represent.

# Funding

We gratefully acknowledge the financial support provided for this project by the Strategic Impact Evaluation Fund (SIEF) of the World Bank and the Education Sector Development Trust.

# Appendices

Appendices are available online at http://bit.ly/asimriazappendices.

# Dedication

The study is dedicated to the living memory of Mr. Muhammad Nouman Bashir, Managing Partner of Weitek Group, Pakistan, the implementation support firm. Mr. Bashir met an untimely demise during the implementation of the intervention.

2
# 1. Introduction

Engagement of citizens in dialogue, monitoring, and decision-making in public services has been widely adopted since 2000 to address state failure in developing countries (World Bank, 2004; Heller and Rao, 2015). Establishment of village-level committees, meetings between service providers, users, and local officials; and training for citizens in monitoring of service; have been associated with improvements in service delivery in India, Uganda, Kenya and Indonesia, among others, and across health, education and other sectors (Mansuri and Rao, 2013; Bjorkman and Svensson, 2009; Duflo et al., 2015; Pradhan et. al., 2014).

In the case of education, attempts to engage communities have focused primarily on School Management Committees (SMCs)—elected local bodies that create agency and voice for community members, allowing them to have more say in the decision-making process. These committees are typically given control over school grants to respond to immediate needs of the school. Committee members may also be given responsibility to monitor teachers and ensure school-age children attend school. Such reforms have been associated with increased enrollment, reduced dropout, and improvements in school facilities and staffing in countries including Cambodia, Honduras and Mexico (Benveniste and Marshall, 2004; di Gropello and Marshall, 2005; Gertler et al, 2011).

However, other countries continue to face several barriers in strengthening community engagement in public services, including low awareness, high opportunity costs, cultural and social restrictions, and low levels of trust in the state (Abraham and Platteau, 2002; De Grauwe, 2005; Joshi, 2013; Croke et al., 2015). Thus, SMCs have not always proven capable of naturally 3
emerging as credible channels of collective action as envisaged, and in many cases suffer from low participation rates and elite capture as noted in reviews of empirical studies (Barrera-Osorio et al., 2009; Bruns et al., 2011).

Government-sponsored reforms to induce community participation in schools have faced similar challenges in Pakistan. The context for this study is Sindh – the second largest province of Pakistan, which has a dense network of public schools that frequently lack basic facilities. More than half of these are one-teacher schools which are prone to temporary closures. A combination of resource shortages, poverty and weak state capacity results in sub-optimal provision of education, leaving a large majority of rural children out of school (Pakistan Social and Living Standards Measurement (PSLM) survey, 2010-11). Dominance of feudal landlords and high levels of ethnic and caste fragmentation in Sindh frustrate any meaningful attempt to invigorate participation of communities to improve service delivery (Miguel and Gugerty, 2005; Hussain et al., 2013).

School grants were introduced in 2001 to transfer authority to elected SMCs, promoting communities’ interest in managing schools, making teachers accountable, and encouraging parents to enroll and retain children in schools. However, these funds remained largely underutilized, and in certain cases were misappropriated. Transfer of funds alone was inadequate in achieving the intended engagement of communities in the school improvement process. A potential explanation is the lack of a mechanism to induce inclusive, broad-based participation of communities with SMCs and schools. In the absence of such mechanisms, Bardhan and Mookherjee (2000) note, decentralized structures are prone to elite capture.

4
This paper provides evidence from a field experiment on the impact of alternative approaches to bolster engagement in schools by creating an inclusive, community-wide interface for parents to engage with other community members. The tested interventions were designed to maximize scalability and thus were simple, low-intensity, and operated on a short timeline of five months.

The first intervention, MEET, aims to catalyze ongoing, face-to-face dialogue between community members about education by externally facilitating a village-wide meeting.<sup>1</sup> Our second intervention, SMS, tests an alternative low-cost approach to dialogue using Information and Communication Technology (ICT), specifically a text message-based Community Dialogue Platform (CDP). In place of an externally facilitated discussion in a village meeting, SMS supported continuous discussion via text message. The SMS intervention was intended to lower the transaction costs for participation by allowing continuous, anonymous, low-cost dialogue at the convenience of users.

Our third intervention, ELECTIONS-CAPACITY, seeks to reform the existing SMC institution to be more effective and more responsive to the preferences of the wider community. This intervention implements elections of SMC members in community-wide meetings, to bring new members into committees, improve alignment between committee members and the community, and reduce elite capture. We employ a cross-over design with the third intervention, implementing

> 1 Under the Sindh system, the wider school community is considered part of an SMC General Body, which has a deliberative and ratifying role through village-wide meetings held twice per year; while a five-person elected SMC Executive Body carries out the majority of duties stipulated in Government of Sindh, School Management Committee guidelines, 2009. We use _SMC members_ to refer to members of the Executive Body and _village-wide meeting_ to refer to the General Body.

5
it as an additional treatment in almost half of the randomly assigned villages in each of the dialogue interventions.

In recent years, a number of experimental studies have tested the impact of various mechanisms to boost community engagement. Two studies from India suggest that furnishing citizens with information about existing educational institutions through meetings conducted in the presence of village administrators had no impact on school attendance or student learning outcomes (Banerjee et al., 2010); yet when similar information was provided by external facilitators through an innovative medium (scripted video drama), it had a significant impact on teachers’ attendance and on student learning outcomes (Pandey et al., 2009). Cerdan-Infantes and Filmer (2015) find that engaging parents through facilitated village meetings increased awareness of a school management program significantly, as did providing information through text messages.

An experiment in Uganda which provided participatory training to SMC members to monitor school quality had significant impacts on student and teacher attendance, and on learning outcomes, while an alternative less participatory training had smaller effects (Barr et al., 2012). However, other studies in low-income settings have found null impacts from attempts to facilitate communities to monitor school quality (Blimpo and Evans, 2011; Beasley and Huillery, 2015). Pradhan (2014) provided grants to SMCs and a program of either classroom-based training, or more participatory facilitation and training, for the use of the grant. Neither approach led to significant improvements in school learning outcomes. However, a sub-treatment which directly supported linkages between SMCs and village councils led to significant improvements to community contributions and learning outcomes. An additional treatment, which supported elections for SMC members, strengthened the observed effects.

6
To our best knowledge, this is the first study that credibly estimates the causal impact of an inclusive, community-wide dialogue interface and benchmarks the results against the standard treatment of externally-facilitated face-to-face meetings. In addition, like Pradhan et al (2014), we add to the literature by estimating the causal impact of elections of SMC members using a randomized evaluation design.

All three interventions were piloted in 284 villages, containing 479 schools, in three representative districts of Sindh, between January and June of 2013. The interventions were expected to have an impact on several school characteristics symptomatic of systemic problems in education service delivery in Sindh (Asim, 2013): school functioning (the open/closed status of school on the day of the visit), enrollment, availability of teachers, teacher presence, infrastructure (e.g., classrooms, drinking water and toilets), and learning outcomes.

We find evidence of increased access to schooling for both the MEET and SMS interventions. Schools participating in MEET were, on average, 9 percent more likely to be open at endline than those in the control group, but were not more likely to reopen if closed at baseline. By contrast, schools participating in the SMS intervention were not more likely to be open at baseline overall, but were 13 percent more likely to have reopened if closed at baseline.

The SMS intervention was associated with a significant increase in enrollments, specifically in retention of boys in Grades 2, 3 and 4, which suggests a sustained improvement in communities’ demand for education over time which was not achieved by the MEET intervention. There were no significant impacts on girls’ enrollment, which may reflect the depth of cultural barriers to girls’ 7
enrollment, or limitations in access to the mobile phones used for the SMS intervention by female family members.

Both MEET and SMS increased the number of registered teachers at schools, and significantly reduced the proportion of one-teacher schools, potentially contributing to some of the observed impact on school functioning. It is likely that both interventions created conditions where local elites felt pressured to increase the supply of teachers in schools. However, neither MEET nor SMS improved the number of teachers actually present at schools; in fact, teacher absenteeism rates, on average, increased in the treatment groups, which reflects a reduction in effort by existing teachers. This suggests that the interventions may have been inadequate in scope to improve community monitoring of resource use in schools. We do not find any significant impact on the availability of school infrastructure from MEET, and observe substantial impacts from SMS on availability of classrooms which do not attain statistical significance.

The addition of the SMC-focused elections and capacity building intervention led to improvements in pre-primary and Grade 1 enrollment, and to availability of infrastructure. However, it also reduced both the impact of MEET on school functioning, and the impact of the SMS intervention on retention of pupils and number of registered teachers. These results stand in sharp contrast to earlier findings by Pradhan et. al. (2014) where elections strengthened the impacts on learning outcomes. In this particular context, holding of SMC elections appears to have reduced the level of community participation in village-wide meetings; a potential explanation is that the presence 8
of a government education official, the Taluka Education Officer (TEO) overseeing the elections, reduced the perceived independence of the interventions.<sup>2</sup> We do not observe impacts on learning outcomes from any intervention; this is likely a result of the lack of impacts on overall teacher presence and potential compositional effects as a result of improved enrollment.

The rest of this paper is structured as follows. Section 2 describes the experiment design, the sampling framework, and the study timeline. Section 3 discusses the data and estimation strategy. Section 4 presents results. Section 5 discusses the findings. Section 6 concludes.

# 2. Experiment

## **2.1 Context**

Sindh, Pakistan’s second largest province, provides a particularly informative environment to test the impact of our interventions, owing to widespread service failures in access to education and a local political economy which holds back community engagement in local institutions.

Sindh has a feudal political economy in rural areas: 76 percent of families are landless and work as tenants, while less than one percent of rural Sindhi families own landholdings of 50 acres or more (Bengali, 2015). These landlords often dominate economic and political life, exercising

> 2 Community meetings took place in the main settlement of the village, in most cases at/outside the premises of the larger, more central school in the village. While the attendees were walking to the announced venue of the meeting (publicized three days in advance), they were able to observe the presence of external facilitators, and in this case, of the Taluka Education Officer (TEO). The sharp decline in number of attendees observed in the ELECTIONSCAPACITY group appears to be the result of general distrust of government officials.

9
control over land, irrigation, credit and service delivery (Hussain et al., 2013). Sindh is also subject to a particularly large degree of ethnic and caste fragmentation. Villages function as the lowest administrative unit, but are typically comprised of spatially dispersed hamlets, _paros_ , each primarily representing a caste or kinship group.

These feudal landholding patterns and fragmentation have led to the development of a dense network of small public primary schools in rural areas. In 2012, Sindh had 42,114 officially established government primary schools, approximately 1.08 schools per 1,000 inhabitants (Asim, 2013). In Figure 1, we illustrate some typical characteristics of a sample village, Seri, in rural Sindh. Seri has households from 11 different castes living in 12 settlements. A total of five public primary schools serve these settlements. The presence of multiple schools within a single village reflects, in part, the extent of social fragmentation: most settlements contain households of one or two castes. As a result, the social barriers against sending children, particularly girls, to a school outside their home settlement are high (Jacoby and Mansuri, 2011). Public primary schools typically feed 1-2 settlements, with a larger school often located in the main settlement, the traditional hub of the village.

Many public primary schools are not functioning or are temporarily closed. In 2012, 14 percent of Sindh’s public primary schools were found to have been closed for more than six months, or to have no teachers or students registered (Asim, 2013). Of the remaining schools, 54 percent had only one teacher, leading to multiple-grade classrooms and a high risk of temporary closure. Of the schools which are functional, many lack important resources and basic facilities. Seventeen percent of schools in 2012 had no building, with classes taking place outdoors in a hut or a 10
temporary structure. Only 58 percent had a toilet, 51 percent had drinking water and 40 percent had electricity (ibid).

These failures of service delivery reflect how schools function as a mechanism of patronage for Sindh’s feudal elite. In many cases, schools serve more as an extension to a landlord’s estate in a village than as a functioning public service (Gazdar, 2000). School buildings are frequently used for non-educational purposes, and politically-appointed teachers are protected from discipline and rarely attend schools (Babur, 2016); given the high rate of one-teacher schools, these high rates of absenteeism exacerbate school closures.<sup>3</sup> As a result of these governance failures, enrollment remains low in Sindh. According to the PSLM survey, 2010-2011, the Net Enrollment Rate (NER) at primary level (ages 6 to 10) in rural Sindh was 54 percent, compared to 67 percent in rural Punjab. Data from our baseline survey, conducted in three representative districts of Sindh, reveal a similar picture: fewer than half (49.8 percent) of school-age children in surveyed households were enrolled in school, and the majority of those unenrolled had never attended school **(Table 1)** .<sup>4</sup>

> 3 During the period of this study, the inadequacy of schooling facilities was exacerbated by extensive flooding in both 2010 and 2011. The 2010 floods destroyed an estimated nine percent of Sindh’s educational facilities and damaged an additional 19 percent and severely disrupted the livelihoods of about 865,000 Sindh households (Asian Development Bank and World Bank, 2011). The 2011 heavy rains resulted in the extent of damage similar to that of the 2010 floods (Asian Development Bank and World Bank, 2012). Schools which were not damaged were converted into temporary shelters for displaced people, disrupting teaching for several weeks. Overall, 66 percent of schools in the villages sampled for this study reported being affected by the floods at the time of the baseline (EMIS, 2011/2012). 4 Although low enrollment often reflects limitations of demand as well as supply, recent evidence suggests that the latent demand for education in Sindh is high; Barrera-Osorio et al. (2017), in an experimental evaluation, found that provision of schooling through public-private partnership to under-served communities in rural Sindh increased enrollment by 30 percentage points for treatment villages.

11
**Table 1: Enrollment by Grade, Children ages (6-10 years old), 2012**

||Public|Private|Other|total|Percent|
|---|---|---|---|---|---|
|Population 6-10 years||||7416|100%|
|Never attended||||3550|47.9%|
|Attended & unenrolled||||171|2.3%|
|Enrolled|3360|249|86|3695|49.8%|
|Currently enrolled:||||||
|_Katchi_* & Grade1|894|56|21|971|13.1%|
|Grade2|1079|63|29|1171|15.8%|
|Grade3|630|55|22|707|9.5%|
|Grade4|365|34|9|408|5.5%|
|Grade5|231|22|3|256|3.5%|
|_Katchi_* – Grade 5|3199|230|84|3513|47.4%|
|Grade 6 and above|161|19|2|182|2.5%|

_<mark>Source:</mark>_ <mark>Data from Sindh Baseline Household Survey, 2012.</mark> _<mark>Note: Katchi</mark>_ <mark>is the pre-primary grade in which children are admitted in a public primary school.</mark> Learning outcomes are still poor for enrolled children. At Grade 3, only 24 percent of students can read words while only 32 percent can perform basic subtraction. At Grade 5, only 19 percent of students can read full sentences while 35 percent can perform division (ASER, 2015). Gender disparities in learning are large and similar for both English and Math; boys outperform girls by 6 percentage points (ibid).<sup>5</sup> SMCs were established at each school in Sindh in the 2000s as part of a broader agenda of decentralization and community participation in public services. These early SMCs included the head teacher and four elected community members, two of which were parents and two nonparents (village elders). However, these early SMCs had little formal control over school decision-

> 5 In this study’s baseline survey, in learning assessments designed to capture students’ achievement at Grade 1-5 levels, tested children (7-13 years) scored an average of only 27 percent across subjects.

12
making or expenditure and in practice remained largely dormant. In 2006, the Government of Sindh attempted to revitalize SMCs by providing annual grants worth PKR 22,000 (USD 220 equivalent in 2012) to support school improvement activities. The SMC was given control over its allocated funds and was empowered to withdraw these resources to implement activities in accordance with a School Improvement Plan (SIP), developed in consultation with the community. However, concerns persisted over the transparency and utilization of the grant funds. In 2009, as part of further reforms, the government distributed updated SMC guidelines to all public primary schools to avoid any fiduciary and audit concerns. These fiduciary controls, however, reduced the role of SMCs to that of drafting an SIP, maintaining auditable records for procurement of goods, implementing nominal repairs and maintenance, monitoring enrollments and ensuring that children in their community started school at the appropriate age. However, they retained no wider role of engaging the community to improve quality of teaching and learning in schools.

The reforms were successful in reinstating the transfer of SMC funds annually; as of June 2012, over 81 percent of SMC bank accounts were active and received grants from the government (World Bank, 2012). Evidence from the baseline survey for this study, conducted in 2012, suggests that the activity level of SMCs improved as a result of the reforms, but remained low overall: while 91 percent of schools had an SMC, fewer than half of these (45 percent) had held a meeting in the previous six to twelve months, and fewer than a quarter (21 percent) had held a village-wide meeting during the same time. Only 29 percent of head teachers reported that SMC members at their school were chosen by election. Underutilization of funds continued: in two of the three 13
districts in which this study takes place,<sup>6</sup> the schools that fell in the 90th percentile in terms of unutilized SMC funds had PKR 77,500 and PKR 62,000, respectively, each equivalent to at least three years of accumulated funds.<sup>7</sup> Private schools are virtually non-existent in Sindh. Ethnic and caste fragmentation means that, despite the large number of public schools, competition between schools is also not a viable source of pressure for improvement. Hence, there is no market-based _exit_ option available to parents (Hirschman, 1970). In the absence of accountability through school choice, and in an environment marked by low democratic pressure on service providers, the only option for dissatisfied parents is to directly _voice_ concerns with the educators (ibid). This makes Sindh a suitable context for exploring alternative approaches to community engagement.

# **2.2 Sample**

Sampling for the study was done in accordance to the districts: (1) being representative of the overall education profile of rural Sindh; (2) having a sufficient number of schools to meet the statistical requirements of our analysis; and (3) not posing excessive security risk for field teams. Using the PSLM survey, we ranked the 28 districts of Sindh by: i) proportion of adults who ever attended school, and ii) net enrollment rates of primary-age children (5-12 years). Also, using the

> 6 The districts in question were Matiari and Mirpurkhas. See section 2.2 for details of selected study districts. 7 We report cost estimates from School Improvement Plans prepared by SMC members during capacity building under ELECTIONS-CAPACITY; participants were instructed to prepare SIPs to spend all accumulated reserves (Figure A3, in Appendix A). In 2013, as the result of initial findings from the baseline survey and pilot activites, the Government of Sindh revamped the scope of the use of SIP Grants, specifically requiring all SMCs in the province to submit bank statements and members’ composition details to be eligible to receive annual funds. The idea was to stop the accumulation of SIP funds in delinquent accounts and only provide grants to schools with active SMC members and functional accounts. This accumulation of funds at the time of intervention might have contributed to expenditures on classrooms, which cannot be constructed alone from the annual SIP grants.

14
Sindh Government’s Annual School Census (ASC) (2010/11), we ranked districts by size as measured by the number of schools and villages in each district. In consultation with the Government of Sindh, we then chose one large, one medium-sized and one small district, ensuring representativeness of the selected districts along two key education measures.<sup>8</sup> The selected districts were Matiari, Mirpurkhas and Sanghar.

In the absence of an updated population census, the ASC was used to set the population frame for selecting villages.<sup>9</sup> Using probability-proportional-to-size (PPS) sampling, we selected a sample of 550 schools drawn from 377 villages. The administrative data appeared to be unreliable regarding the exact location of schools and the inclusion of schools that are permanently closed. To address this, we mapped 300 of the 377 villages in a village-census exercise. The maximum of 300 villages was reached using a randomized ranking procedure.<sup>10</sup> The process through which treatment schools were selected from the village mapping exercise is outlined in **<u>Figure 2</u>** .

The school sampling strategy for the baseline covered public primary schools that were open on the day of the visit or closed for a period of less than one year prior to the day of the visit. We sampled all such schools, identified by the local community, in the main settlement of our sampled

> 8 Matiari was ranked the third smallest district, Mirpurkhas was ranked 12th, and Sanghar was ranked 18th, according to the number of schools and villages in each district. In terms of education indicators, Mirpurkhas had one of the lowest levels of education outcomes, Matiari was in the middle, while Sanghar was among the highest. They were deemed relatively safe by the government for field teams to visit.

> 9Given the heterogeneity in village size, we employed Probability Proportional to Size (PPS) sampling to create selection probabilities proportional to the number of students enrolled in primary education in a village. Virtually every village in Sindh has a government public primary school.

> 10 In order to select 300 villages from 377 (per our evaluation design), a village mapping firm identified and listed all schools in each village in the rank-order specified, until they hit the maximum sample of 300 villages. They were allowed to skip villages only when i) schools were not found in listed revenue villages due to noisy administrative data; or ii) field-teams were denied permission by local residents to enter the village. However, the number of such cases was small, with total replacements amounting to fewer than 20 villages.

15
villages; and an additional 15 percent of schools, in other settlements within each village.<sup>11</sup> Our total sampling consisted of 501 schools across 296 villages. This is representative of the study districts, and of rural Sindh, in general, across a range of key indicators.<sup>12</sup> However, the sample differs significantly from study districts in the number of female teachers per school, and from all of rural Sindh in school enrollment and availability of drinking water; these indicators are linked to heterogeneity in population, as well as location-specific endowments.<sup>13</sup> Following data collection and analysis, the final sample of the study is 284 villages containing 479 schools.<sup>14</sup> Of these, 387 schools in 249 villages were consistently open at baseline and endline. Therefore, measurement and analysis of outcomes of interest were restricted to this sample, except the school status on an unannounced visit. Schools that switched status from closed to open at endline are excluded, as the characteristics of these schools are largely different from schools that were consistently open both at baseline and endline.<sup>15</sup>

# **2.3 Treatments**

All interventions employed a similar structure and institutional arrangements. In all treated villages, a local community mobilization firm visited villages and announced a community-wide

> 11 In villages that did not have any school in the main settlement, a maximum of three schools were sampled from other settlements based on their total enrollment. For villages that did not have either a functional or temporarily closed school in any of its settlements, we included all schools, 16 in total, even if they were closed for a period of more than one year.

> 12 See Table A2, in Appendix A.

> 13 See Table A2, in Appendix A.

> 14 Field teams were not allowed to enter four villages at the time of baseline. In addition, eight villages had to be dropped at the data analysis stage for incompleteness and inconsistencies in collected data. 15 See Table A3, in Appendix A.

16
meeting via loudspeakers, posters and conversations with residents.<sup>16</sup> Various dissemination practices were pre-tested in non-sample villages to identify the most appropriate and effective ways to mobilize the community for village-wide meeting. At the meeting, participants completed an attendance sheet and provided a mobile phone number. They then watched a performance by local schoolchildren and listened to a specially-prepared 10-minute audio clip presenting a dramatized story which promoted the importance of education and provided basic information on SMCs, their structure, purpose, and membership, as well as the amount of funds available for their use. The clip and delivery instructions were developed by an international communication firm, in consultation with stakeholders, to maximize effectiveness.<sup>17</sup> The content of the remainder of the meeting, and any follow-up activities, varied according to the intervention.

In the first intervention, MEET, we test the impact of ongoing face-to-face dialogue around education and schools through an externally facilitated village-wide meeting. Following the introductory activities described above, meeting attendants were introduced to another audio clip, a 20-minute dramatized discussion highlighting specific actions community members could take to improve education outcomes in schools. The audio clip emphasized the need for parents and the broader community to take an active interest in the schooling of their children by participating in further village-wide education meetings and visiting the school regularly to check the presence and activity of teachers, monitor the use of SIP grants, and note any need for repairs and

> 16 There was no placebo meeting or sensitization efforts done in control communities. All treatment groups’ findings therefore include the impact of these common sensitization activities, which were not carried out in control villages. 17 Based on a feedback cycle, the audio clips were revised multiple times, from the original, poem-based format, and finally to an audio drama. An international marketing and communication firm, JWT Worldwide, was engaged to develop the story plot in drama format. The design team also sought to make the language of the audio clips easy to understand, and to adapt the clip to the context of rural Sindh. Facilitators were provided with timed scripts to ensure consistent implementation of the meeting.

17
maintenance. A moderated discussion was then held among meeting participants around key ideas communicated in the audio script. The discussion was designed to serve as a village-wide meeting on school issues as required by the SMC guidelines. At the end of the discussion, existing SMC members were introduced to the villagers and their names and phone numbers given to participants. Participants were encouraged to independently organize and conduct follow-up meetings twice a year as stipulated in the guidelines.

In the second intervention, SMS, we assess the impact of an alternative ICT-based approach to facilitation of dialogue, employing a specially-designed text message-based Community Dialogue Platform (CDP). Following the standard introduction as described above, facilitators played a customized audio clip which introduced the CDP, its purpose and core functions, followed by a hands-on demonstration by field facilitators.

The CDP was designed to offer a similar combination of information and facilitated discussion as MEET, in virtual form. First, key information shared in the MEET intervention, regarding rights and responsibilities and the functions of the SMC and SIPs, was sent through SMS messages to the community members.<sup>18</sup> Second, participants were provided with a free number to which to send comments and responses to questions regarding conditions at the local school. Users could send messages consisting of any comment or text; these were then categorized according to topic based on key words used. Participants could send any number of messages. Through this, the CDP elicited preferences and concerns about education from community members.

> 18 See Figure A2, in Appendix A, for sample messages.

18
A key aspect of the CDP was to combine this _elicitation_ of preferences with _diffusion_ of opinions to all registered participants on the system. Over the five months following the meeting, participants received messages on a weekly basis summarizing the topics and content of the messages received by the system. This was intended to enable users, even those who did not send messages to the platform, to be kept informed about dialogue going on through the platform about local schools and of each other’s main concerns. It was envisioned that salient issues would then be further discussed by community members in informal gatherings.

The selection of an SMS-based platform for the intervention was informed by the high rate of mobile phone penetration in Pakistan – 72 percent in mid-2013 (Pakistan Telecommunication Authority, 2014) – which provided a low-cost and accessible way to engage communities. The design of the system was intended to reduce barriers to participation: messages were anonymous, in order to minimize any personal risk or consequences to participation for users, and airtime was provided to participants to defray the cost of sending messages. It was anticipated that these low barriers to participation would mean that activity on the CDP would be sustained throughout the five-month period of operation.<sup>19</sup> A potential threat to the effectiveness of an SMS-based approach was the low level of literacy in Sindh <mark>: only 60 percent of males above the age of 10 in rural Sindh, and only 22 percent of females, were literate prior to the start of our interventions (PSLM 2010-11). This was addressed through two methods. First, volunteers, selected by communities, were appointed to support</mark> mobile-

> 19 The pilot intervention was funded through Education Sector Development Trust Fund, and the allocated funds were available only for FY-13. The intervention started in January with completion date of June 30, 2013.

19
illiterate members in contributing to the system. Volunteers could send messages on behalf of users with literacy or other barriers and receive airtime credit as compensation. About five percent of messages sent to the CDP were sent by these volunteers on behalf of other users, in addition to other users who sent their own messages with the support of volunteers. Second, in order to ensure that all users could participate in the CDP in their own language, messages were made available in both Sindhi and Urdu. An automated voice call was used to inform users that they had a choice of language and to collect their preference.

The third intervention, ELECTIONS-CAPACITY, targeted the SMC institution directly by supporting fresh, transparent elections of SMC members, to introduce new members with greater enthusiasm and align SMCs with village preferences, and participatory capacity building to support newly elected members to fulfil their roles.

The elections took place at the end of the introductory village meeting. Taluka Education Officers (TEOs), government sub-district education officials, were required to attend meetings and provided with an honorarium to officiate elections. Neither the attendance of the TEO, or the fact that elections would be conducted were announced in advance; however, anecdotal evidence suggests that the arrival of the TEO in the village typically alerted residents to their intention to attend the meeting. It was intended that TEOs would afford official legitimacy to the election and provide letters to enable the newly elected SMCs to assume official authority and claim SIP grant funds. We find no evidence of any interference from the TEO in the proceedings of elections or the transfer of charge to newly elected members.

20
Following the elections, the SMC members were provided with hands-on training during three structured meetings conducted over a three-week period. The meetings familiarized members with the functions of the SMC and provided participatory training on how to develop an SIP and present it at the village-wide meeting for ratification. In addition, members were trained on how to withdraw and manage grant funds and oversee implementation.

# _Allocation to treatment_

The evaluation follows a clustered randomized controlled trial (RCT) for causal identification of the impact of three interventions on school-level outcomes. Villages are randomly assigned to one of the treatment groups or the control. The unit of randomization is the village and the school is the unit of inference.

In a cross-over design, the third intervention is combined with each of the dialogue interventions in a randomly selected subset of schools, creating four distinct treatment combinations. **Table 2** summarizes the structure of the interventions and treatment groups.

**Table 2: Cross-over Experimental Design**

|||**Dimens**<br>|**ion 1:**<br>||
|---|---|---|---|---|
|||No facilitation of<br>dialogue|Meeting-based<br>community dialogue|SMS-based<br>community dialogue|
|**Dii 2**|No SMC elections and<br>capacity building|Control|MEET|SMS|
|**menson :**|SMC elections and<br>capacity building||MEET+ELECTIONS|SMS+ELECTIONS|

21
# **2.4 Theory of change and anticipated impacts**

It was anticipated that the MEET and SMS interventions could impact schools through several channels **<u>(Figure 3)</u>** . First, it was expected that the information provided on the importance of education through the MEET and SMS interventions would lead to increased salience of education within communities, and demand for schooling by parents. Second, the information provided, within both interventions, on the community’s role in school management was anticipated to improve the level of effort by communities to monitor the use of resources within schools. MEET was especially intended to demonstrate the benefits of holding meetings, and communities were expected to hold those meetings twice a year.

Third, it was anticipated that the elicitation and diffusion of local knowledge and preferences carried out in both MEET and, in particular, SMS could lead to increased social pressure by communities on local elites to improve the quality of local schools. We postulate that a platform that encourages a multiplicity of opinions through anonymous exchanges, and diffusion of preferences among all members, vitiates elite control over framing of public opinion on quality of service delivery in villages. We anticipated elites would respond to community-identified needs, favorably, mobilizing more resources to school in a bid to secure their status as the ones perceived as working for the benefit of the community.

It was expected that the ELECTIONS-CAPACITY intervention, which focused on SMCs, would improve the effort of SMC members and their focus on the priorities of the wider community, and improve the utilization of SIP grants.

22
We identify five key anticipated impacts on schools from the interventions: school functioning; availability of teachers and teacher attendance; school infrastructure; enrollment; and learning outcomes.<sup>20</sup> _School functioning:_ We anticipate impacts from community engagement on school functioning as a response by local elites and school administrators to the increased salience of and demand for education. Such a response could be the result of the removal of barriers, e.g. the misappropriation and misuse of school buildings (Gazdar, 2000); in the case of Sindh, where lack of teachers is a key contributing factor in school closures, it could also reflect increased supply of teachers, specifically a reduction in the number of schools with only one teacher.

_Enrollment:_ We anticipate impacts on school enrollment because of increased parental interest in education, and in the case of ELECTIONS-CAPACITY, from improvement in SMC efforts to enroll out-of-school children. We also anticipate greater retention of students because of improvements in school quality as perceived by parents.

_Availability of teachers_ and _teacher attendance:_ As described above, we anticipate improved provision of teachers to schools as a response by elites and authorities to deliberation on education issues within communities. In addition, we also anticipate improvements in teacher attendance because of enhanced monitoring of school resource use by communities.

> 20 The pre-analysis plan was not registered in a secure independent register in 2011 when the experiment was designed, as is the best practice today.  However, a concept note was prepared and approved by the Strategic Impact Evaluation Fund (SIEF).

23
_Infrastructure:_ We anticipate possible improvements in school infrastructure as a result of increased effort by SMC members. Although the annual figure of US$ 220 equivalent (2012) for SIP grants is insufficient for large infrastructure investments, in schools with accumulated carryover funds it was expected that increased community engagement, coupled with the election of new committee members, would result in substantial spending to address infrastructure gaps, particularly shortages of toilets and classrooms in schools. Furthermore, improvements in simple infrastructure such as toilets could reflect increased contributions of labor by both SMC members and other community members.

_Learning outcomes:_ We anticipate potential improvements in learning outcomes primarily as a result of improvement in teacher availability and reductions in teacher absenteeism. However, given the expectation of improved enrollment, we also anticipate a potential reduction in average learning outcomes because of compositional change in the student body (Crawfurd, 2018).

# **2.5 Randomization**

**<u>Table 3</u>** shows balance checks between treatment groups on key indicators at baseline. The treatment and control groups are balanced in terms of baseline statistics along a range of covariates and outcome variables. None of the treatment groups differs significantly from the control group in any indicator.<sup>21</sup>

> 21 We run a regression of each treatment dummy on all variables and test the joint significance of all coefficients using an F-test. We fail to reject the null and conclude that there is no difference between any groups.

24
# **2.6 Treatment fidelity**

We first measure the share of adults in a village attending the community-wide meetings, which provided the forum for facilitated discussion under MEET, introduction of the CDP under SMS, and the electing of SMC members under ELECTIONS-CAPACITY **(Table 4)** .<sup>22</sup> Overall, on average the participation rate was 63 percent among treatment villages. Mirpurkhas had the highest participation rate with 75 percent of households, followed by 57 percent in Sanghar and 51 percent in Matiari. These district-level differences in participation rates are controlled in the proceeding estimations using district fixed effects.

The MEET and SMS groups have higher participation rates, on average 67 percent and 68 percent respectively, compared to 58 percent for MEET+ELECTIONS and SMS+ELECTIONS. This suggests that some aspect of the ELECTIONS-CAPACITY intervention actually reduced attendance at village meetings. The differences in attendance are particularly stark for Matiari district, where average participation rates were only 38 percent in groups receiving ELECTIONSCAPACITY, and 63 percent for MEET and SMS treatments alone.

For the SMS intervention, the other key aspects of treatment fidelity are registration and participation on the CDP platform. A total of 4,981 unique users registered on the CDP portal. The average registration rate in SMS villages (65 percent) was higher than that in SMS+ELECTIONS

> 22 The population was estimated from the household census conducted for the listing exercise. The take-up rate is measured as the number of attendees as a proportion of the number of adult individuals residing in the main settlement of each village. The project teams were instructed to revisit any village for which attendance was under 20 percent and hold a second meeting. Reported attendance is for the second meeting in these villages. As discussed in Section 2.2, the sample includes all main settlement schools and 15 percent of schools outside main settlements of sampled villages.

25
villages (61 percent). All registered users received weekly messages summarizing the opinions expressed by participants anonymously on the platform. Therefore, dialogue via text message encompasses not only active messages sent by informed community members, but also passive receipt of summary messages by all registered users.

A smaller proportion of users provided inputs into the system: 28 percent of registered users, 1,229 in total, sent at least one non-junk message into the portal during the course of the project.<sup>23</sup> Input into the portal reduced in frequency over the course of the operation of the CDP, but activity continued throughout the implementation phase: an average of 16 messages per village were sent in the ten days following the introductory meeting; four messages per village were sent during days 31-40 of operation; and two per village during the last ten days of implementation five months later.<sup>24</sup> This decline in sending of messages was expected as, once the set of problems in a particular village was identified, e.g., shortages of teachers and classrooms, and sent to all users in a weekly summary, it was unlikely that other users would repeatedly voice the same concern(s). Rather, the expectation was that these issues would be subject to continued discussion informally within the village.

For the first key aspect of the ELECTIONS-CAPACITY intervention, the holding of elections for SMC members, participation was almost universal: elections took place in all villages except for two, both in SMS+ELECTIONS, one in Mirpurkhas and the other in Sanghar, where the TEO was not available. We also measure the extent to which the membership of SMCs changed following

> 23 Incomplete, or off-topic, or blank messages (i.e. not concerning education) were labeled as junk. 24 See Figure A4, in Appendix A.

26
elections. The findings suggest that composition change was large, if not total: the average change was 73 percent, equivalent to almost three new members; all four elected members were replaced in slightly more than one-third of schools (36 percent). This was measured during the school endline in 2015, demonstrating that the changes in SMC composition were sustained for a period of at least two years.

We also measure the attendance rate at the capacity building for SMC members, the second key aspect of the ELECTIONS-CAPACITY intervention. The mean attendance for the three meetings, in both MEET+ELECTIONS and SMS+ELECTIONS schools, was 96 percent, and members from all schools in ELECTIONS-CAPACITY groups participated.

Overall, fidelity to treatment was sufficiently consistent across treatment groups to enable accurate assessment of impact of interventions on key outcomes. Variance in take-up between districts does pose a potential threat to internal validity; this is addressed by the inclusion of district fixed effects.

# 3. Data and Estimation Strategy

## **3.1 Data**

Data for this study come from a village-level household census, and school and household surveys conducted both at baseline and endline. First, a census of households and schools was conducted in three study districts in 2011, which provided a population frame for random selection of villages and schools. Second, a baseline survey was conducted in schools and sampled households between 27
April 2012 and January 2013.<sup>25</sup> Third, an endline survey was conducted in all schools from January 2015 to March 2015,<sup>26</sup> and in households between January and June 2017.<sup>2728</sup>

**<u>Figure 4</u>** <u>summarizes the experiment sampling frame and timeline.</u>

In both the baseline and endline, the school surveys collected detailed data on school-level variables such as pupil enrollment and attendance, school infrastructure, facilities and resources, the school SIP, and SMC functioning and membership. The questionnaire also collected data on the total number of teachers employed in school and a count of those present in school on the day of the visit. The visits were held on a day when schools were supposed to be open but were not announced in advance. Information on enrollments, teacher and student presence, and SMC members was collected from school records in interviews with the head teachers. In addition, for the endline round of school surveys, a random sample of 20 students from Grade 3 and 4 from each school were interviewed and given standardized tests for Math, English and Sindhi.

In addition, household surveys for both baseline and endline gathered information on household demographic and socioeconomic characteristics, household schooling choices and engagement

> 25 The initial plan was to collect all baseline data in 2012. However, due to heavy flooding in Sindh province, the data were collected in two rounds from April to June 2012, and November-December 2012. The flooding delayed implementation of the pilot activities but did not significantly disrupt data collection beyond this delay. We control for a flood dummy in all regression estimates.

> 26 The endline survey exercise included two phases: surprise visits to schools in January 2015, and announced visits made to schools in the sampled villages between January 2015 and March 2015.

27 Following analysis of the school endline survey data, the endline household survey was conducted for a sub-sample of 160 randomly selected baseline villages. Power calculations and budget limitations motivated the decision to reduce the endline household sample. Initial visits were conducted at the household level in January 2017 to obtain tracking data on household members and children, with the full survey conducted from March to June 2017.

28 The World Bank worked with local survey firms: Research Consultants (RCons) and Weitek Group on various rounds of data collection. The baseline and endline surveys were conducted by Weitek and RCons, respectively, to mitigate any risks associated with biased reporting in interviews by the same firm for both rounds.

28
with SMC members from the male and female head of the household. A student questionnaire was administered to sampled children regardless of school-going status. Standardized tests were administered at this time to primary school-age children (ages 7-13), for Math, English and Sindhi.<sup>29</sup>

# **3.2 Outcome variables**

As described in section 2.2, the pilot interventions could potentially impact a number of outcomes: school functioning; student enrollment; availability of teachers; teacher attendance; school infrastructure; and learning outcomes. In order to ensure accuracy and adherence to the study timeline, we measure all variables through directly observed survey data.

For school functioning, we measure whether schools were open at the unannounced visit at baseline and endline.<sup>30</sup> Since a number of schools switched status between the visits, it is important to capture the direction of change from open to closed and vice versa, in relation to the treatments. This generates three specific outcome indicators: number of schools open; number of schools open at baseline which were closed at endline; and number of schools closed at baseline which had reopened at endline.

> 29 Standardized tests for English, Math and Sindhi were administered to children ages 7-13 at the household level both for the baseline and endline. The instruments were designed, pre-tested and piloted based on students’ textbooks and curriculum standards for Grades 1-5 of primary school. We fitted a two-parameter item response theory (2pl-IRT) model for English and Math scores to account for what students of similar knowledge and skills ought to know, by distinguishing between difficulty level (see Appendix B for details). Sindhi tests include open-ended question items and hence an IRT score cannot be fitted.

> 30 If a school was found closed, verification was sought from community members. If a school was closed for one or two days only, a revisit was made when the school was reopened and the school was not recorded as having been closed. If a school was confirmed by community members to have been closed for some time, and/or was found closed on both of two revisits, it was recorded as closed.

29
For enrollment, we first measure the number of pupils enrolled in _katchi –_ pre-primary – and Grade 1, according to school enrollment records at both baseline and endline. To measure the retention of students into upper grades, we also measure the number of pupils enrolled in the ensuing grades 2-5, using the same records.

For school infrastructure, we measure change in the number of classrooms, and whether a school has any of four key facilities that are important for a safe, secure and comfortable school environment: toilets, electricity, drinking water, and a boundary wall.<sup>31</sup> For availability of teachers, we first measure the average number of teachers registered at schools at endline. In addition, in recognition of the fact that the large number of one-teacher schools appears to be a central driver of school closures in Sindh, we also present the share of schools which have only one registered teacher. For teacher attendance, we measure the number of teachers present at schools during unannounced visits at baseline and endline<sup>32</sup> ; as well as the share of teachers absent during these visits, in comparison to the number registered at the school.

For learning outcomes, we measure IRT scores in Mathematics and English, based on the standardized tests described above.<sup>33</sup> We report findings from tests administered to children

> 31 Electricity, sanitation and water are particularly important given the prevalence of high temperatures in Pakistan, where temperatures during summer term periods can often reach upwards of 40°C, while a wall is important for student safety and to support student attendance (Andrabi et al., 2007).

> 32 Teacher presence and absence was measured at both surprise and announced visits; data from the unannounced visit were used except in 29 cases where the school was closed at the unannounced visit but open at the announced visit. The results are robust if restricted to the unannounced sample only.

> 33 The Sindhi test has a large number of open-ended questions. Only multiple-choice items were used to fit the 2plIRT model. Results of the multiple-choice section of Sindhi were largely consistent with the English and Math tests and are omitted for brevity.

30
tracked between baseline and endline at household level; as a robustness check, we also report cohort level results for Grade 3 and 4 students tested at the school level.

# **3.3 Mediating variables**

Very few studies have tested any structural model to identify the specific channel through which institutions supporting community engagement impact education outcomes. Nonetheless, we report some additional statistics on mediating variables which can be measured robustly from the detailed household and school level surveys to describe possible ways the interventions might have impacted outcomes.

To assess monitoring of school resources by the community, we analyze data on the frequency with which household heads report having visited schools in academic years 2015/16 and 2016/17.<sup>34</sup> We also present descriptive statistics from the endline household survey on the key priorities which household heads discuss with head teachers when they visit school.

To measure the engagement of communities with SMCs, we employ data from household surveys at baseline and endline testing the awareness of household heads of (i) the existence of the SMC, (ii) its membership, (iii) the frequency of its meetings, (iv) the amount of school grants it controls, and (v) the requirement to prepare a SIP.

To measure improvement in alignment of SMC activity with communities’ priorities, we analyze the main items of expenditure in SIPs developed during the capacity building exercise provided to

> 34 A greater number of visits to the school or to see the Head Teacher suggests a greater interest in education, as well as more engagement in the way resources are utilized at the school level. We note that the number of visits to school may not fully capture the quality or relevance to school issues of interaction between teachers, head teachers and parents at the school level.

31
SMCs as part of ELECTIONS-CAPACITY, in comparison with the main topics of discussion in SMS messages received by the CDP.

# **3.4 Descriptive Statistics**

**<u>Table 5</u>** shows the observed values at endline in the control group villages and the various treatment groups, for a range of indicators.

The findings suggest that, absent the intervention, the problems of the school system in Sindh have not considerably improved. In terms of school functioning, only 80 percent of schools in the control group were found to be open at the unannounced visit at endline. In terms of enrollment, the average school had 71 students in grades from _katchi_ (pre-primary) to Grade 5, suggesting the continuation of Sindh’s general pattern of multiple small schools. Only 46 percent of students at control schools at endline were female, suggesting continuing gender disparities in enrollment.

Infrastructure remained an area of significant shortage in control schools. The average control school had 2.2 classrooms available for use at endline. Sixteen percent employed at least one openair classroom. Toilets appear to be an area of significant shortage: only 75 percent of control schools had toilet facilities available, with an average student-toilet ratio of 59:1. Only 51 percent of control schools had electricity, 60 percent drinking water, and 70 percent a boundary wall.

In terms of staffing and teacher attendance, the control endline data suggest no significant improvement absent the intervention. The average control school had 2.25 teachers<sup>35</sup> ; an average

> 35 The number of teachers in control schools has not decreased as a result of the interventions. We believe that any impact on new teachers in treatment schools is from assignment of teachers to these schools, and not resulting from

32
five percent of teachers were absent even during announced visits. Fifty-two percent of control schools remained one-teacher schools at endline.<sup>36</sup>

# **3.5 Estimation strategy**

We estimate the school level effect of intervention (intent-to-treat or ITT effect) by fitting Analysis of Covariance Model (ANCOVA): where is an outcome variable at endline for school _i_ in village _v_ and district _d,_ and ii,vv,dd,e ddeeiie vv is the vector for village-level treatment indicators. YY controls for the baseline value of TT ii,vv,dd,b bbe iie outcome variable to improve the precision of the point estimate. In addition, the vector of controls XX include school size, and a flood dummy to account for any lagged effect of exogenous shock to schools before the intervention. The term denotes district-level fixed effects which control for dd any district-level differences in participation rates. We adjust the standard error to account for uu within-village correlations across schools in outcomes.

The coefficient of interest is , which indicates the magnitude of the effect of each treatment, MEET, SMS, MEET+ELECTIONS and SMS+ELECTIONS, with respect to the control group. ββ

> other teachers taken away from nearby communities (controls) and transferred to the treatment schools. 36 In terms of learning outcomes, test scores remained low. School-age children undergoing learning assessment at the household level at endline in control villages scored, on average, 22 percent across subjects, compared to 36 percent for the cohort tested in schools. In the Household sample we include children who have never attended school but are in the school-age (7-13).

33
We test the sensitivity of our estimates to inclusion of treatment interaction with participation rates at the village meeting.<sup>37</sup> ANCOVA is the preferred estimator, as the standard difference-in-difference (DiD) estimator would require twice as much sample to yield the same power when baseline is taken, and autocorrelations are low (McKenzie, 2012).<sup>38</sup> To measure the effect of treatment status on student test scores we fit the following ANCOVA model: where, is the test score for student _k_ in subject _m_ in household _t_ in village _v_ at the kk,mm,t,v,e ddeeiie follow-up. SS is the indicator variable for treatment assignment. controls for the vv kk,m,tt,v,b bbe iie baseline value of the same students’ test scores. The term TT denotes district-level fixed effects. XX dd We adjust the standard errors to account for within-village correlations across students in uu outcomes. We test the sensitivity of our estimates to inclusion of treatment interaction with participation rates at the village meeting.

# **3.6 Attrition**

There is some attrition in each round of data collection. We used the sample of 287 villages and 489 schools for randomization. For a handful of villages, the data quality was poor, with

> 37 We are unable to find any significant interaction effects with participation rates and any outcome of interest. Results are omitted for brevity, but available on request.

> 38 The study is underpowered to detect small effect sizes for a standard DiD estimator.

34
incomplete surveys returned at the endline, leading to missing data for 1% of study villages. We report pre-intervention characteristics for 284 villages in **Table 3** . In addition, the survey teams could not collect data on 71 schools and 58 schools in the baseline and endline round of surveys; these schools were temporarily closed and despite repeated attempts data could not be collected from these schools. As shown in Table A4,<sup>39</sup> there are no systematic differences in attrition by treatment group. The difference-in-differences among attritors and non-attritors, across schools in treatment and control groups, reported in the last column, suggests that there is no indicator with a statistically significant value for difference-in-differences. We also test for joint significance of individual coefficients being equal to zero using seeming unrelated regression (SUR), the resulting Chi square statistic suggests we should not reject the null that both samples are balanced.<sup>40</sup> While attrition is not systematically different across treatment and control groups, but the diminished sample size nonetheless may have reduced the precision of the estimates.

# 4. Results

## **4.1 Average ITT effects: Community mobilization and facilitation of dialogue**

This section presents the impact of the first two interventions, MEET and SMS, both of which focused on facilitation of community dialogue and deliberation on school issues among community members. These results are shown in the first two estimations in each of **Tables 7-11.** In addition to presenting ITT estimates in comparison to control, we also carry out equivalence testing (F-stat) to directly compare the outcomes in MEET and SMS. Where relevant, we also

> 39 See Appendix A.

> 40 We use a similar test as applied for attrition by Barrera-Osorio et al, 2018.

35
report findings from simple comparison of means between treatment groups at endline to motivate the results **<u>(Table 5).</u>** For certain outcomes of interest, we also conduct transition probability analysis, measuring the likelihood of a change in school-level outcomes **<u>(Table 6)</u>** <u>.</u><sup>41</sup>

# **4.1.1 School functioning**

On average, the MEET intervention increased the number of schools that were functional in the treatment group as compared to control **(Table 5)** . To understand the direction of change in school functioning between baseline and endline, we conduct transition probability analysis of the likelihood of the school changing status from closed to open between baseline and endline; to capture any potential reduction in school closures, we also capture the likelihood of schools changing from open to closed **(Table 6)** .

The results demonstrate that schools in MEET were significantly less likely than control schools to close between baseline and endline. 4.3 percent of MEET schools closed, compared to 13.3 percent in the control group; this is equivalent to a 67 percent reduction in the likelihood of schools closing. However, we do not observe a significant improvement from MEET in the likelihood that schools transitioned from closed to open.<sup>42</sup>

> 41 The analysis is directional, assigning a value of zero if no or negative change occurred in a given outcome at a school and one if positive changed occurred. The implicit assumption in transition analysis is that any negative impact on the outcome variable is the result of factors unrelated to the intervention. The difference in transition percentages between each treatment group and the control represents the additional likelihood of a change in outcomes in the stated direction.

> 42 Schools were found closed on the day of visit for either baseline or endline. The limitation of the analysis is that the population of schools that switched status is a small subset of intervention schools and we do not have the full set of covariates to test the balance between treatment and control groups for this subset. The share of schools found closed on unannounced visit at baseline is balanced across treatment groups, giving a degree of confidence that changes in school status are largely driven by the intervention.

36
For SMS, the picture is more mixed. The intervention did not significantly reduce the likelihood of a school closing between baseline and endline; however, the intervention does appear to have led to reopening of schools which were closed at baseline. For these schools, the chance of transitioning from closed to open were more than double that for control schools: 22.3 percent versus 10.2 percent, which is an improvement by a magnitude of 1.2 times.<sup>43</sup> Next, we present the regression estimations for school functioning **<u>(Table 7)</u>** .<sup>44</sup> The findings reinforce those of the comparison of means and the transition probability analysis. MEET schools were 11 percent more likely to be open at the unannounced visit at endline than control schools; they were also significantly (9 percent) less likely to have changed from open to closed. SMS schools were not more likely to be open at endline, but if closed at baseline, were significantly (13 percent) more likely to have reopened.

# **4.1.2 Enrollment**

**<u>Table 8A</u>** presents overall and gender-specific impacts on school-wide enrollment. The MEET intervention does not appear to have significantly improved enrollments for either male or female students; however, we do observe a large and significant impact on boys’ enrollment from the SMS intervention, with schools having an average 10 more male students at endline than control

> 43 We note methodological limitations of the finding as school status changes are limited to 102 schools in the study sample which were closed either at baseline or endline. We are cognizant of the fact that a small number of villages might be contributing to the observed impacts. Nonetheless, flipping of school status at a much higher rate than the controls is an important effect to be captured given the context of this study. Reduction in the share of one-teacher schools in treatment groups further gives credence to the effects observed on the school functioning indicator. 44 All regression estimates control for district fixed effects, to rule out impacts from variation in participation rates between districts; as well as a flood dummy which controls for whether a school had been affected by the floods in the period before baseline, in order to rule out bias from flood-related impacts on outcome variables.

37
group schools. This is a large increase, given the small school sizes common in Sindh, equivalent to additional enrollment of 21 percent. Equivalence testing confirms that the SMS effect on total boys’ enrollment is significantly larger than that of MEET. No significant impact was observed from either intervention on girls’ enrollment.

**<u>Table 8B</u>** presents enrollment impacts by grade, specifically for male students. No significant impact is observed from either MEET or SMS on _katchi_ or Grade 1 enrollment, suggesting a lack of effect of the interventions on the likelihood of new school-age boys being admitted to school. However, significant impacts are observed from SMS on boys’ enrollment in Grades 2, 3 and 4, suggesting improvements in retention of male students throughout the school cycle because of the intervention. SMS schools had an average 2.2 additional students per grade in Grades 2-4 than control schools, equivalent to an increase in enrollment in these grades of 29 percent. The SMS impacts were significantly larger than those of MEET in Grades 2 and 3.

**<u>Table 8C</u>** presents impacts on girls’ enrollment. No significant impacts are observed in any grade.

# **4.1.3 Availability of teachers**

Turning to registered teachers, comparison of means **(Table 5)** suggests significant impacts from MEET and SMS on the proportion of schools with only one teacher: one-third of MEET schools (32 percent) and SMS schools (35 percent) had only one teacher at endline, versus more than half (52 percent) of control schools. Transition probability analysis also suggests significant impacts from both MEET and SMS on the total number of teachers available at schools: MEET schools were almost twice as likely as control schools to gain at least one additional teacher; 29.7 percent 38
of SMS schools gained at least one teacher **(Table 6)** —the addition of ELECTION to MEET and SMS did not lead to any further improvement.

A similar pattern is observed in the regression point estimates **<u>(Table 9)</u>** <u>. MEET and SMS schools</u> were both significantly less likely than control schools to be single-teacher schools at endline. MEET schools had 0.26 additional teachers at endline compared to control schools, although the result is not statistically significant<sup>45</sup> ; SMS schools had an additional 0.3 teachers, significant and equivalent to a 13 percent improvement in staffing.

# **4.1.4 Teacher attendance**

In terms of the number of teachers actually present at a school, we do not observe any impact from MEET on point estimates **(Table 9)** . Findings for the share of teachers absent confirm that the rate of teacher absence was significantly higher (6 percentage points) in MEET schools, a 120 percent increase relative to the control group mean of 5 percent, suggesting that the gains in staffing were offset by increased absenteeism.

Turning to SMS we do observe a substantial positive impact on teacher presence from the intervention, with treated schools having an average 0.24 additional teachers present at endline compared to control, equivalent to a 12 percent increase in teacher presence; however, the standard

> 45 The study is sufficiently powered to identify policy-relevant effect size of 0.2 standard deviation or more. However, given a context of very small school sizes with limited number of teachers and very limited infrastructure, it is important that future studies sufficiently increase the number of observations to allow detection of small effect sizes, particularly for teachers and infrastructure variables.

39
error is large, and the result is not significant. The rate of teacher absence was higher in SMS schools than in control, but the difference is statistically insignificant.

# **4.1.5 Infrastructure**

Turning to physical infrastructure, **Table 6** also presents transition probability comparisons for the addition of classrooms and toilets. In the comparison of means **(Table 5)** , we include estimations for other key school facilities: electricity, drinking water, and boundary walls at schools. We find no significant impacts from either the MEET or SMS intervention on any infrastructure outcome in either estimation.

In the point estimates **<u>(Table 10)</u>** <u>, we include all infrastructure variables. We find no impact from</u> MEET or SMS on any infrastructure variable which is statistically significant in comparison to control. However, in the case of SMS, a substantial impact is observed on the number of classrooms, with 0.23 additional classrooms at SMS schools; this is significantly greater than the level in MEET in equivalence testing.

# **4.1.6 Learning outcomes**

Finally, we present findings on the impact of the interventions on learning outcomes from standardized tests conducted during household surveys. Comparison of means **(Table 5)** shows no differences in test performance between any treatment group and the control for scores. **<u>Table 11</u>** 40
shows regression point estimates at the student level.<sup>46</sup> We find no significant impact on learning outcomes for either the MEET or SMS interventions.<sup>47</sup>

# **4.2 Average ITT effects: Support to elections and capacity of SMC members**

The third and fourth estimations in each of **Tables 7-11** present the impact of the MEET and SMS interventions combined with the ELECTIONS-CAPACITY intervention.

# **4.2.1 School functioning**

We do not observe significant positive impacts on school functioning from the ELECTIONSCAPACITY intervention. In fact, the addition of the intervention appears to have had a negative impact which offsets the impact of the MEET and SMS interventions. In the comparison of means **(Table 5)** and transition probability analysis **(Table 6)** , the impact of MEET in reducing the likelihood of school closures is smaller and no longer significant in MEET+ELECTIONS schools. Similarly, the impact of SMS in increasing the likelihood of schools reopening is reduced and no longer significant in SMS+ELECTIONS schools. The same pattern is observed in the point estimates **(Table 7)** . However, the impacts of SMS+ELECTIONS on schools reopening, while not significantly different from control, are significantly larger than the impacts of MEET+ELECTIONS in equivalence testing.

> 46 We also conduct estimations at the village level and find no difference in results.

> 47 The findings are subject to two potential limitations: the passage of time since the interventions, and the negative impact of out-of-school children. To address these, we also present estimates based on the same tests during school surveys, conducted with enrolled students only in 2015; we again observe no significant impacts from either treatment (Table A5, in Appendix A).

41
# **4.2.2 Enrollment**

A similar pattern is observed for measures of school-wide enrollment **(Table 8A)** . As with school functioning, the impacts observed from the SMS intervention on overall school enrollment and boys’ enrollment are reduced and become marginally insignificant in SMS+ELECTIONS. However, turning to impacts on grade-wise enrollment **(Table 8B)** , SMS+ELECTIONS appears to have had significant impacts on enrollment of _katchi_ and Grade 1 boys. The average SMS+ELECTIONS school had four more students in these grades than in the control group, an impact of 21 percent. No significant impacts are observed in girls’ enrollment **(Table 8C)** .

# **4.2.3 Availability of teachers and teacher attendance**

We find no significant positive impacts from ELECTIONS-CAPACITY on availability of teachers, and the intervention appears to have reduced the impact of MEET and SMS. In the comparison of means **(Table 5)** and point estimates **(Table 9)** , the impact of MEET and SMS on the likelihood of a school having only one teacher is reduced and no longer significant in MEET+ELECTIONS and SMS+ELECTIONS. In the point estimates, in contrast to SMS, no significant impact from SMS+ELECTIONS on teacher numbers is observed in the regression analysis. Similarly, in the transition analysis **(Table 6)** , the impacts of MEET and SMS on the likelihood of a school gaining additional teachers are reduced and no longer significant in both MEET+ELECTIONS and SMS+ELECTIONS.

As with MEET and SMS, no significant impact is observed from either MEET+ELECTIONS or SMS+ELECTIONS on the number of teachers present at school **(Table 9)** . The share of teachers absent was significantly higher in SMS+ELECTIONS.

42
# **4.2.4 Infrastructure**

In the case of physical infrastructure, in the transition probability analysis **(Table 6)** , more than one in four SMS+ELECTIONS schools – 27.7 percent – gained an additional classroom, versus 17.6 percent of SMS schools and 15.1 percent of control schools. Furthermore, 21.9 percent of SMS+ELECTIONS schools gained an additional toilet, versus 16.4 percent of control schools and 16.2 percent of SMS schools.

In the point estimates **(Table 10)** , as with SMS, the impact of SMS+ELECTIONS on classrooms was large but not statistically significant, but SMS+ELECTIONS was again found to have had a positive impact on the availability of toilets, increasing by ten percent the share of schools with at least one toilet. Similar to MEET, no significant impact is observed on any infrastructure indicator from MEET+ELECTIONS in any estimation.

# **4.2.5 Learning outcomes**

As in the case of MEET and SMS, we find no significant impacts on learning outcomes in English and Math from either MEET+ELECTIONS or SMS+ELECTIONS in either comparison of means **(Table 5)** or point estimates **(Table 11)** .

# **4.3 Average ITT effects on household awareness of the SMC**

To explain the extent to which the intervention succeeded in improving engagement between communities and SMC members, we measure household awareness of the SMC and its functions and processes. **<u>Table 12</u>** <u>presents impacts from the various interventions on the awareness of</u> household heads about the existence of the SMC; its membership; the frequency of its meetings; the amount of SIP grants; or the requirement to complete an SIP. We find no significant impacts 43
from any of the interventions on any awareness measure. The evidence suggests that the intervention did not raise community awareness of the existence, functioning, or purpose of SMCs. This finding is in line with evidence from India suggesting that the provision of information alone frequently fails to raise community awareness of the functioning of existing community engagement institutions (Banerjee et al., 2010).

# **4.4 Threats to internal validity**

Using geospatial analysis (GIS), we find that five percent of intervention villages are located 1-2 kilometers from the nearest control village.<sup>48</sup> This proximity may endanger the validity of the impact estimates if contamination of treatment occurred in control schools and villages.<sup>49</sup> In order to test that contamination did not occur, we isolated all treatment schools in villages located at least 1 kilometer from control schools, and we conducted the analysis on this limited sample to identify any significant changes in the size of coefficients and standard errors. The results on additional teachers, teacher absenteeism, and primary school enrollment for boys all continue to be statistically significant.<sup>50</sup> In addition, the results remain consistent in magnitude compared to the main results tables, with negligible changes in effect sizes, suggesting that contamination has not biased the impact estimates reported within the full sample and fully functional sample of schools.

> 48 The mean distance between a treatment school and the nearest control school is 7.41km; the median 5.83km, and the standard deviation 12.67km.

> 49 The Stable Unit Treatment Value Assumption (SUTVA) requires that the treatment of one unit should be unaffected by the assignment of treatment to other units (Wooldridge, 2012). The relatively close distances between control and treatment schools heighten the probability of contamination, a possible violation of the SUTVA assumption. 50 See Table A6, in Appendix A.

44
# 5. Discussion

## **5.1 School functioning and availability of teachers**

Our findings suggest that facilitation of community-wide dialogue around education can succeed in catalyzing both demand for, and the supply of, schooling. Both MEET and SMS had significant impacts on school functioning. Specifically, while MEET had a predominantly defensive impact on school functioning by preventing closures, SMS was effective in causing closed schools to reopen. Point estimates suggest that MEET reduced the likelihood of schools closing by 9 percent, while SMS improved the likelihood of closed schools reopening by 13 percent.

MEET and SMS both significantly reduced the share of one-teacher schools; SMS also significantly improved the overall number of teachers available at schools by 13 percent. These results suggest that both SMS and MEET created conditions where local elites felt pressured to increase the supply of the most important resource, teachers, in schools. Shortage of teachers was one of the most frequently discussed topics on the CDP (see **Figure 5A** , in section 5.3). In terms of the mechanism, the finding that household awareness of the functioning of the SMC did not increase suggests that SMC members were not the primary drivers of these results. We believe that general discussion facilitated by the interventions led to a more concerted effort by the community to work with village elders to lobby authorities for additional teachers.

The addition of ELECTIONS-CAPACITY, which sought to reform the SMC into an institution that improves school outcomes through better engagement with wider community, instead appears to have undermined the positive impacts observed for MEET and SMS on school functioning, student retention and staffing. The results suggest that the intervention undermined the 45
effectiveness of community dialogue around education. A potential explanation is that the inclusion of a government official, the TEO, at the village-wide meeting where elections were held had the unintended effect of reducing community trust in the new SMC, and overall community engagement over time. We find that participation in the village-level meetings in SMS+ELECTIONS and MEET+ELECTIONS villages was around ten percentage points lower than in MEET and SMS villages. Neither the holding of elections nor the the participation of the TEO was announced during village mobilization activities prior to the meeting; however, residents walking to the meeting may have turned back once they saw the TEO approaching the venue. This reflects the lack of public trust in the government functionaries in Sindh, as well as popular perception of widespread failures of service delivery.<sup>51</sup> Another potential explanation is that the election of new SMC members compromised the ability of the SMC to mobilize resources for schools. The replacement of existing SMC members, often appointed by elites, with transparently elected members may have damaged networks and informal linkages effective in delivering some degree of inputs to schools; this is in line with findings from Indonesia that strengthening linkages between SMCs and powerful village councils had the largest impacts on learning outcomes (Pradhan et al., 2014).

The finding suggests that, in conditions of low trust in government, trade-offs may exist between the twin goals of strengthening community engagement and supporting elections of representative local bodies. This is in line with other studies in South Asia which suggest that government-led

> 51 The fact that attendance was lowest in Matiari district provides further support for this interpretation. Matiari was carved out of a larger district, Hyderabad, in 2005. The expectations were high that the creation of a small district with dedicated line authorities would improve the functioning of services. The failure to meet these expectations might be another reason for evidently low confidence afforded to the civil administration officers in this particular district.

46
efforts to bolster community engagement are less effective than those implemented by external, independent actors (Pandey et al., 2009; Banerjee et al., 2010).

# **5.2 Enrollment**

The ELECTIONS-CAPACITY intervention, when combined with SMS, succeeded in driving improvement in the intake of new children to schools ( _katchi_ and Grade 1 enrollment). The most likely mechanism of impact is an improvement in effort by the SMC, driven by new members, to identify out-of-school children and enroll them in schools. To some extent the continuity of dialogue on the CDP may also have offset the loss of community interest resulting from the TEO’s presence in the village-wide meeting. In MEET+ELECTIONS, however, owing to the one-off nature of the discussion, such complementarity effects are absent, and there was no possibility of regaining trust of the community once the village-wide meeting was over.

The SMS intervention also improved the retention of students, as measured through enrollment in Grades 2-4, with point estimates suggesting gains of 29 percent. This suggests a sustained improvement in communities’ demand for education over time which was not achieved by the one-off MEET intervention. The improvement in enrollment may also reflect improvement in the perceived quality of schooling in the eyes of parents, in response to the improvements in availability of teachers achieved by the SMS intervention. In SMS+ELECTIONS schools, where improvements in teacher availability were not significant, we do not observe significant gains in retention despite the improvements in _katchi_ and Grade 1 enrollment.

The impacts of the SMS and SMS+ELECTIONS interventions on enrollment and retention did not extend over to female students. This may simply suggest that the interventions, although effective, 47
were not strong enough to overcome cultural barriers to girls’ enrollment. Alternatively, this may suggest that the ICT-based approach of the CDP did not overcome barriers to female participation. If women are more likely to respond to efforts to improve girls’ enrollment but were limited in their ability to access CDP on phones controlled by male heads of household, it follows that girls’ enrollment would not benefit significantly from the SMS intervention (Gigler and Bailur, 2014). Given the much lower rate of literacy among females than males in Sindh, an additional explanation is that the efforts made as part of the CDP to address literacy through the use of volunteers were not successful to engage female participants.<sup>52</sup>

# **5.3 Infrastructure**

The MEET intervention does not appear to have driven improvements in infrastructure, either with or without the addition of ELECTIONS-CAPACITY. As in the case of availability of teachers, this suggests a one-off meeting did not provide a sufficiently sustained improvement in community engagement to drive improvement in infrastructure. However, we do find some evidence of impacts from the SMS intervention on school infrastructure. The addition of ELECTIONSCAPACITY appears to have strengthened these impacts. The findings suggest that the ongoing community engagement engendered by the SMS intervention catalyzed effort by newly elected SMC members and led to improvement in school facilities identified by communities. In order to test this interpretation, we analyze the content of messages received by the CDP **<u>(Figure 5A)</u>** ; the messages demonstrate a clear preference by communities for investment in physical infrastructure,

> 52 It is worth noting that the majority of primary schools in Sindh are mixed-gender, in contrast to other provinces of Pakistan. The share of sampled schools which were girls-only was too small to enable robust assessment of whether impacts were observed on enrollment in these schools in particular.

48
the single largest item accounting for 38 percent of messages received. Analysis of the expenditure priorities identified by SMC members during the participatory capacity building exercise carried out as part of ELECTIONS-CAPACITY **(Figure 5B)** reveals a similar priority placed on physical infrastructure, with building, classrooms and toilets accounting for a total of 34 percent.

# **5.4 Teacher attendance and learning outcomes**

Despite improvements in the availability of teachers, none of the interventions significantly improved the number of teachers actually present at schools at endline. Comparison of means **(Table 5)** demonstrates that the share of teachers absent at endline was actually greater in MEET and SMS schools, suggesting that gains in teacher availability were balanced out by greater absenteeism. This is in line with findings from India and Kenya, which suggest that increased staffing at schools is often counteracted by a reduction in effort and presence by existing teachers (e.g. Muralidharan and Sundararaman, 2013; Duflo et al, 2015).

The result suggests that the interventions were not successful in catalyzing improved community monitoring of resource use in schools. Data from household surveys at endline reveal a high level of community engagement with schools: more than one-third (34.5 percent) of households reported having met the head teacher of their child’s school on at least a monthly basis in the 2015/16 school year, two years after the intervention. However, we find no systematic evidence of greater levels of school visits in either SMS or MEET schools than in control schools.

Furthermore, analysis of households’ reported concerns about their children’s schools reveals that teacher absence was not considered a significant problem by households, suggesting that community engagement may be of limited impact if misconceptions persist about the severity of 49
problems in service. More generally, the findings are in line with those from Niger (Beasley and Huillery, 2015) that communities are unlikely to successfully monitor school resource use without significant investments in capacity.

Neither MEET nor SMS had any significant impacts on learning outcomes. This finding is consistent with the lack of observed impact on teacher effort with no improvement in teachers’ presence at school. It is also possible that, while individual students did experience some benefit in learning, these effects were offset by the impacts of increased retention of lower-performing students, reducing average performance; this is aligned with Crawfurd (2018) where the authors find null effects on learning from contracting-out of public schools in Punjab, Pakistan to a private company, despite significant impacts on access.

# **5.5 Cost effectiveness**

**<u>Figure 6</u>** presents the implementation cost for all three interventions: MEET, SMS and ELECTIONS-CAPACITY. Costs for both MEET and SMS were similar, totaling US$129,059 for MEET and US$138,369 for SMS. Taking the total endline enrollment of treated schools as the population of impact, this suggests a per-student cost of US$16.26 per student for MEET and US$19.41 for SMS. With 25 percent of expenditures invested at the design stage, per-student expenditures for ongoing and scaled-up activities are estimated to be significantly lower: US$12.85 for MEET and US$15.66 for SMS.

Taking this variable cost as a starting point, **<u>Table 13</u>** presents cost effectiveness estimates of the intervention. Improvement in enrollment is achieved from two intervention impacts: increase in enrollment in schools which remained open at baseline and endline, and additional enrollment 50
achieved through the opening of closed schools. For every $100 spent on operational costs, SMS intervention contributed to 1.07 additional children enrolled in school compared to 0.91 for MEET. These estimates are net of improvements in enrollment in the control group and are adjusted by scaling-up the gains in control group to the equivalent number of schools in each treatment.

# **5.6 Limitations of analysis**

Using an ANCOVA estimation strategy assumes that the treatment and control villages exhibit statistical balance not only on baseline characteristics, but also in treatment take-up. We therefore conduct robustness checks in which we include variables that could potentially cause village-level heterogeneities in treatment take-up, including a village-level asset index (proportion of individuals who do not have any land), basic literacy scores for males and females, and cellphone penetration rate.<sup>53</sup> Any technology- or information-based intervention such as SMS exhibits a literacy and gender bias: take-up is likely to be concentrated among educated males. If mothers are the family members most likely to influence girls’ enrollment, his skewed take-up might partly affect why primary enrollment increased only for boys and not for girls. To correct for literacy bias, the treatment includes the installment of a village volunteer who works as a liaison to send text messages for the less-literate and technologically challenged villagers. As mentioned previously, we also find no change in our estimates when we control for literacy scores. Gender bias is not directly addressed by our interventions, however, any future studies can focus on treatments targeted at women to

> 53 Regression estimates omitted for brevity, but available on request.

51
test whether women are able to affect policy making and public service delivery, and also to see how their social preferences may differ from those of men.

# 6. Conclusion

This paper studies the impact of three treatments to test simple and low-intensity approaches to bolster community engagement in schools: community-wide dialogue at an externally facilitated meeting; continuous dialogue via text messages, on a high-frequency, anonymous and low-cost text message platform; and elections and training of SMC members. We find that measures aimed at creating community-wide, inclusive platforms to promote dialogue lead to greater interest of the community in education with large impacts on access and staffing. However, attempts to reform an existing institution, the SMC, in a cross-over design, had limited effect. In most cases, it undermined the effects of dialogue interventions.

Despite the simple and low-intensity nature of the intervention, and comparatively short period of implementation (five months), we find both SMS and MEET to be largely successful in improving availability of schooling by increasing the number of open schools and enabling closed schools to reopen, and increasing the availability of teachers. SMS also proved effective in improving retention of boys throughout the primary school cycle. We suspect these impacts arise from anonymity of exchange and continuity in dialogue. However, we caution policy makers to carefully consider the time, effort and investment that is needed for such an ICT infrastructure to be developed, deployed and iteratively adapted to respond to local needs. Even in this controlled experiment, several parameters had to be meticulously calibrated with the local firm to make the ICT platform work. We expect future iterations of the intervention to study more carefully the 52
gender divide in take-up of the intervention and impacts. Technology, in our case, appears to have reproduced the gender inequities both at the stage of engagement with the platform, as well as in outcomes, on access to schooling for girls.

Neither MEET nor SMS improved the number of teachers actually present at schools, reflecting a reduction in effort by existing teachers – this reflected the low numbers of parents who recognize teacher presence as a serious issue. We also do not observe impacts on learning outcomes, which may reflect the lack of impacts on teacher presence; or compositional effects on the student body as a result of improved enrollment.

The addition of the ELECTIONS-CAPACITY intervention produced benefits for Grade 1 boys’ enrollment, and for availability of toilets and classrooms, suggesting increased effort by newly elected SMC members; however, it reduced the impacts of both MEET and SMS on school functioning, overall enrollment, and availability of teachers. The intervention appears to have reduced participation in village-wide meetings, potentially the result of public resistance to the presence of a local education official.

The field experiment raises some interesting questions for future research work. The significant decline in attendance at community-wide meetings which included the TEO, the lowest government administrative functionary, summarizes in a snapshot a possibly broken relationship of citizens with the state. The mere presence of a state official at the time of the village meeting may have reduced the credibility of the intervention among community members. Further research is required to more fully illuminate the ways in which trust in government affects the relative 53
effectiveness of community engagement mechanisms incorporating a greater or lesser degree of official involvement.

54