metadata: authors: 0: author: Nyqvist, Martina Bjorkman
1: author: Svensson,Jakob
2: author: De Walque,Damien B. C. M.
count: Uganda
docna: 0: docna: Information is power : experimental evidence
            on the long-run impact of community based monitoring
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Reproductive Health,Early Child and Children's Health,Health Service Management and Delivery,Law and Justice Institutions
teratopic: Social Development,Industry,Communities and Human Settlements,Health, Nutrition and Population
abstracts: cdata!: This paper presents the results of two
            field experiments on local accountability in primary health
            care in Uganda. Efforts to stimulate beneficiary control,
            coupled with the provision of report cards on staff
            performance, resulted in significant improvements in health
            care delivery and health outcomes in both the short and the
            longer run. Efforts to stimulate beneficiary control without
            providing information on performance had no impact on
            quality of care or health outcomes. The paper shows that
            informed users are more likely to identify and challenge
            (mis)behavior by providers and as a result turn their focus
            to issues that they can manage locally.
geo_reg_key: 517191
repnme: repnme: Information is power : experimental
            evidence on the long-run impact of community based monitoring
display_title: Information is power : experimental
            evidence on the long-run impact of community based monitoring
listing_relative_url: /research/2014/08/20144947/information-power-experimental-evidence-long-run-impact-community-based-monitoring
topicv3: Infant and Child Mortality m1326544 1446,Health Workforce m1326573 631,Health Services Delivery m1327294 1305,Mortality m1504365 3661,Health m1504661 3480
publishtoextweb_dt: 2014-08-01T00:00:00Z
historic_topic: Social Development,Industry,Communities and Human Settlements,Health, Nutrition and Population
no_of_pages: 53
author_keys: 0: author_key: 125655
1: author_key: 687876
model_extractions: WPS7015 Policy Research Working Paper 7015

# Information is Power

Experimental Evidence on the Long-Run Impact of Community Based Monitoring _Martina Björkman Nyqvist Damien de Walque Jakob Svensson_ Development Research Group Human Development and Public Services Team August 2014
Policy Research Working Paper 7015

# **Abstract**

This paper presents the results of two field experiments on local accountability in primary health care in Uganda. Efforts to stimulate beneficiary control, coupled with the provision of report cards on staff performance, resulted in significant improvements in health care delivery and health outcomes in both the short and the longer run. Efforts to stimulate beneficiary control without providing information on performance had no impact on quality of care or health outcomes. The paper shows that informed users are more likely to identify and challenge (mis)behavior by providers and as a result turn their focus to issues that they can manage locally.

This paper is a product of the Human Development and Public Services Team, Development Research Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at martina.bjorkman.nyqvist@hhs.se ; ddewalque@worldbank.org and jakob.svensson@iies.su.se.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# Information is Power: Experimental Evidence on the Long-Run Impact of Community Based Monitoring<sup>*∗</sup>

Martina Björkman Nyqvist<sup>†</sup> Damien de Walque<sup>‡</sup> Jakob Svensson<sup>§</sup> August 2014 _Keywords:_ Community Driven Development, Health, Information, Participation JEL classification: I11, I15, O15

* We are deeply indebted to Frances Nsonzi for her contributions at all stages of the project and acknowledge the important contributions of R. Reinikka, G. Kajubi, A. Ojoo, A. Wasswa, J. Kanyesigye, P. Okwero, R. Chase, C. Winter, I. Njosa, M. Bitekerezo, and the field and data staff with whom we have worked over the years. We thank Andrea Guariso for excellent research assistance. We thank the Uganda Ministry of Health, the World Bank's Country Office in Uganda, and the Social Development Department, the World Bank, for their cooperation. We are grateful for comments by Adam Wagstaff and seminar and conference participants at Dartmouth, University of Michigan, Monash University, the World Bank, DIAL conference in Paris, and the NEUDC conference at Harvard. Financial support from the BNPP, the World Bank Research Committee, the Gender Action Plan (GAP) and the Strategic Impact Evaluation Fund (SIEF) at the World Bank, the Swedish Research Council (421-2009-2209) and the Program for Development Research, SIDA is gratefully acknowledged.

> † Stockholm School of Economics and CEPR. Email: martina.bjorkman.nyqvist@hhs.se

> ‡ The World Bank, Development Research Group. Email: ddewalque@worldbank.org

§ IIES, Stockholm University and CEPR. Email: jakob.svensson@iies.su.se
# **1 Introduction**

Poor quality plagues public service provision in many developing countries.<sup>1</sup> In response, policies to enhance beneficiary involvement as a way of strengthening demand-responsiveness and local accountability are becoming increasingly popular. Despite the enthusiasm for this approach, however, the evidence provides mixed results about its short-run effectiveness.<sup>2</sup> Whether it can lead to sustained improvements in service provision is largely unknown.<sup>3</sup> First, we provide evidence of the longer run impact of a local accountability intervention in primary health care provision in Uganda. The _participation & information intervention_ combined the dissemination of report card information on staff performance with efforts to enhance participation. Four years after the initial intervention, we document significant improvements in health care delivery (increased utilization and improved adherence to clinical guidelines) and health outcomes (reduced child mortality and increased weight-for-age and height-for-age for children), in the treatment as compared to the control group. Thus, properly designed, enhanced beneficiary involvement can result in large and sustained improvements in both health service provision and health outcomes.

Second, we shed light on why this particular community-based intervention resulted in such a large and sustained change in service provision while several other seemingly similar interven-

> 1Das et al. (2008) in their overview found quality of care to be very low in many developing countries. For example, doctors in Tanzania completed less than 25% of the essential checklist for patients with malaria, a disease that is endemic in the country. Indian doctors asked an average of one question per patient (“What’s wrong with you?”). Chaudhury et al. (2006) found an average absence rate of 27% among primary school teachers and 37% of primary health center staff in Uganda. Primary students in urban schools in Tanzania spend about one quarter of the daily schedule in a classroom where the teacher is actually present. Roughly half of the primary school teachers in Senegal fail to demonstrate mastery of the curriculum their students are supposed to master. Every other primary health clinician in Senegal is unable to detect a simple case of pneumonia and in total spends about half an hour per day counseling patients (Bold et al, 2011).

> 2Olken (2007) finds that grassroot participation in the monitoring of a village road construction program in Indonesia had little average impact. Banerjee et al. (2010) show that a project giving local committees in India the tools to evaluate student performance had no effect. Casey et al. (2012) evaluate an infrastructure project in Sierra Leone involving both relatively large grants and the application of processes to enhance local empowerment and participatory governance in the planning and implementation phases. They do not find any evidence that the intervention led to fundamental changes in collective action at the village level. Duflo et al. (2012), on the other hand, find that a governance program that gave parents specific training on how to monitor and assess teachers’ effort and performance in a contract-teachers program in Kenya resulted in significant improvements in learning. Björkman and Svensson (2009) find that providing users with information on health workers’ performance and facilitating the development of an action plan resulted in significant improvements in both health workers’ performance and health outcomes in Uganda. For a review of the World Bank’s experience with local participatory development programs, see Mansuri and Rao (2013).

3While lack of evidence of the long-run impact is not specific to studies on beneficiary control, it is of particular concern here since public service providers may easily adjust their behavior for a short period of time given the large pre-existing X-inefficiencies in public service provision, but permanent changes in behavior may be harder to institute. Moreover, interventions aimed at increasing community participation are primarily meant to influence norms and collective actions and thus influence local decision making both in the short and the longer run.

1
tions have not had much of an impact. Our focus here is on the role and impact of information. Specifically, we designed an intervention (the _participation intervention_ ) replicating the participation but not the information component of the _participation & information intervention_ . Most community empowerment projects focus on participation and the _participation intervention_ can be viewed, from a process point of view, as being fairly representative of the standard Community Driven Development (CDD) approach promoted by the World Bank and other donors. Holding context constant, we evaluate whether differences in intervention design – the provision (or not) of information on the health clinic’s performance – can help explain the mixed findings in the literature.<sup>4</sup> The impacts of the interventions with and without information differ markedly. Without information, the process of stimulating participation and engagement (the _participation intervention_ ) had little impact on the health workers’ behavior, health outcomes or the quality of health care. In contrast, when community members are informed; i.e., have objective and quantitative information about staff behavior, the same type of processes resulted in significant improvements in health care delivery and health outcomes in both the short and the longer run.

Using data from the implementation phases of the two interventions, we investigate why the provision of information appears to have played such a key role. A core component of both experiments was the agreement of a joint action plan outlining the community’s and the providers’ agreement on what needs be done, and by whom, in order to improve health care delivery. While the process of reaching an agreement looks similar on some observable measures in the two treatment groups – the same number of community members participated in the community meetings and, on average, the two groups identified the same number of actions to be addressed – the type of issues to be addressed differed significantly. In the _participation group_ , the health provider and the community identified issues that primarily required third-party actions; e.g., more financial and in-kind support from upper-level authorities and NGOs. In the _participation & information group_ , by contrast, the participants focused almost exclusively on local problems, which either the health workers or the users could address themselves, including absenteeism, opening hours, waiting time, and patient-clinician interactions. These results are consistent with the hypothesis that lack of information on performance makes it more difficult to identify and challenge (mis)behavior by the provider, and hence constrains the community’s ability to hold providers to account. That is, with access to information, users are better able to distinguish between the actions of health workers and factors beyond their control and, as a result, turn their focus to issues that they can manage and work on locally.

Taken together, our results provide both encouraging, and less encouraging, news for those pro-

> 4As pointed out in Banerjee et al. (2010), from the available evidence, it is difficult to disentangle whether the mixed findings are driven by differences in the details of the intervention or context.

2
moting greater beneficiary control. We find that an intervention that largely mimics the Community Driven Development (CDD) approach; i.e. the _participation intervention_ , had little impact.<sup>5</sup> The technically more demanding (and more expensive) _participation & information intervention_ , however, resulted in a more engaged community and large and long-run improvements in both health service provision and health outcomes.

The next section describes the institutional setting for our study. Section 3 details our evaluation design, the features of the two interventions, and the data used to evaluate them. The long-run impact evaluation results of the _participation & information intervention_ are presented in section 4, while the evaluation results of the _participation intervention_ are discussed in section 5.<sup>6</sup> Section 6 presents some suggestive findings in support of the key mechanism, relates our findings to the mixed results in the literatures and discusses the policy recommendations of our findings.

# **2 Institutional setting**

The experiments were implemented in 75 rural communities served by a public primary health facility (or dispensary) in nine districts covering all four regions of Uganda. Dispensaries are in the lowest tier of the health system where a professional interaction between users and providers takes place. Most dispensaries are staffed by 6-10 workers – an in-charge or clinical officer (a trained medical worker), nurses, nursing aids and other assistants – and according to the government health sector strategic plan, the standard for dispensaries includes preventive, promotional, outpatient care, maternity, general ward, and laboratory services (Republic of Uganda 2000). Health services should be provided for free.

The health sector in Uganda is decentralized and a number of agents are responsible for supervision and control of the dispensaries. At the lowest tier, the Health Unit Management Committee (HUMC) is supposed to be the main link between the community and the facility. However, the baseline data reveal that these institutions, including the HUMC, are not actively involved in the supervision or support of primary health care providers.

The setting for our experimental study – rural Uganda – is characterized by poor public health service provision (large X-inefficiencies). For example, roughly 50% of the staff are absent from the clinic on a typical day (based on observational data from unannounced visits); the average waiting time is more than two hours; and only four out of ten patients report that any equipment

> 5The CDD approach has become one of the most common approaches to poverty reduction (broadly defined) in developing countries. Over the past decade, the World Bank alone has allocated close to $85 billion to local participatory development programs (Mansuri and Rao, 2013).

> 6The short-run evaluation results of the _participation & information intervention_ are presented in Björkman and Svensson (2009).

3
was used the last time the respondent (or the respondent’s child) visited the clinic.<sup>7</sup>

# **3 Experimental design and data**

## **3.1 Overview**

The research project was initiated in 2004 and extended in 2007. Of the 75 rural communities and health facilities, 50 facilities/communities were included in the first phase of the project in 2005 (the _participation & information intervention_ ) and 25 facilities/communities were added in 2007 (the _participation intervention_ ). The catchment area or community for each dispensary was defined as the households residing in the 5-km radius around the facility. A community in our sample does, on average, have 2,500 households residing within the five-kilometer radius of the clinic, of which 350 live within a one-kilometer radius.

For the _participation & information_ experiment, the units (facility/community) were first stratified by location (districts) and then by population size. From each block, half of the units were randomly assigned to the treatment group (25 units) and the remaining health facilities were assigned to the control group. A similar procedure was initiated in 2007 when the project was extended with the _participation intervention_ ; i.e., after stratifying on location and population size, the 25 new facilities were randomly assigned to a treatment group (13 units) and a control group (12 units).

Trial sizes were set to detect effects on utilization and under-five child mortality. The trial sizes were also influenced by logistical and cost constraints and the anticipation of smaller long-run treatment effects. In each community we surveyed approximately 100 households (and collected birth and death statistics from approximately 100 under-five children). Thus, the sample for the _participation & information_ experiment consists of 50 communities/health facilities and approximately 5,000 households. The sample for the _participation_ experiment consists of 25 communities/health facilities and approximately 2,500 households. Mortality and utilization data are available for 2005 and these data were used to estimate study power (Björkman and Svensson, 2009). The estimated overall under-five mortality rate (the number of child deaths per 1000 child-year observations) in the control group in 2005 was 34.1. Assuming 300 child-years of observations in each cluster (three years, 2006-2008, and 100 child observations per year), 50 clusters and an estimated coefficient of variation of 0.15, the _participation & information intervention_ has 80%

> 7While we have no data on the clinicians’ workload, other estimates from similar settings suggest a low workload. For example, combining observational data on time spent per patient with facility records on the number of patients treated per day, Bold et al. (2011) estimate that the total time spent counseling patients per day per clinician in rural primary health clinics in Tanzania and Senegal is only 26 minutes.

4
[60%] power of detecting significant differences at the 0.05 significance level if the intervention reduced mortality by 25% [20%]. Assuming 200 child-years of observations in each cluster (two years, 2007-2008, and 100 child observations per year), 25 clusters, and a coefficient of variation of 0.15, the _participation intervention_ has 80% [60%] power of detecting a significant difference at the 0.05 significance level if the intervention reduced mortality by 40% [32%].

The mean utilization in the control group in 2005 was 660 with a standard deviation of 175. The _participation & information intervention_ therefore has 80% [60%] power to detect a significant difference at the 0.05 significance level if the intervention increased utilization by 21% [17%]. The _participation intervention_ has 80% [60%] power to detect a significant difference at the 0.05 significance level if the intervention increased utilization by 29% [23%].

As a reference point, Björkman and Svensson (2009) show that the _participation & information intervention_ reduced under-five mortality by an estimated 35% and increased utilization by 20-29% after one year. Thus, while the power to pick up significant treatment effects in the _participation intervention_ is lower than that of the _participation & information intervention_ , both experiments are powered to pick up effects of similar size as those reported in Björkman and Svensson (2009).

# **3.2 Interventions**

The aim of the research project was twofold. First, to evaluate the long-run impact of the community monitoring intervention initiated in 2004 (the _participation & information intervention_ ). Second, to assess what role and impact dissemination of quantitative information on the staff’s performance has in these types of beneficiary control programs. To this end, we designed a new intervention in 2007 – the _participation intervention_ .

Efforts to stimulate beneficiary involvement, like the Community Driven Development (CDD) approach, operate on the principles of local empowerment and participatory governance as mechanisms to strengthen demand-responsiveness and local accountability. The core of the strategy is the process through which problems and constraints are identified and how (local) decisions are made and executed. While there are variations across projects, in practice community driven development is achieved though facilitated meetings. Both the _participation intervention_ and the _participation & information intervention_ largely followed this approach. The key difference between the two interventions is that the _participation & information intervention_ also included the dissemination of a report card with data on the health clinic’s performance in various dimensions.

The research design allows us to estimate and compare three treatment effects, holding the context; i.e., health care provision in rural Uganda, constant:

- The short-run treatment effect of the _participation & information intervention_ (reported in Björkman and Svensson, 2009);

5
- The long run **,** i.e., four years after the initial intervention, treatment effect of the _participation & information intervention_ ;

- The short-run treatment effect of the _participation intervention_ .

The design is illustrated in figures 1 and 2. Note that while the long-run evaluation covers the period 2005-2008, the short-run treatment effects are estimated over two consecutive time periods (2005 and 2007-2008, respectively). Holding context and timing constant, we can compare the long run (4 years) treatment effects of the _participation & information intervention_ with the shorter run (2 years) treatment effects of the _participation intervention_ and holding context constant, we can compare the short run effects of two interventions.

# **The participation intervention**

The _participation intervention_ involved three types of meetings facilitated by staff from local community-based organizations (CBOs): (i) a community meeting – a two-day afternoon meeting with community members from the catchment area and from all spectra of society and with on average more than 150 participants per day and per community attending; (ii) a health facility meeting – a half-day event, usually held in the afternoon at the health facility, with all staff attending; and (iii) an interface meeting – a half-day event with representatives from the community and the staff attending.<sup>8</sup> The main objective of these meetings was to encourage community members and health facility staff to develop a shared view of how to improve service delivery and monitor health provision in the community; i.e., to agree on a joint action plan or a community contract. In practice, the process of reaching an agreement was achieved in two steps. First, in the community and health facility meetings, using various participatory methods, including focus group discussions, community score cards, and role plays, the community and the health staff were asked independently to identify and prioritize the key problems and issues they viewed as the most important to address and how to address them within the current resource envelope.<sup>9</sup> Second, in the interface meeting, representatives from the community and health facility staff presented and discussed their suggestions and in a facilitated meeting, agreed on a set of issues that were viewed as most critical to address. These issues were put into a joint action plan. The action plan outlines the community’s

> 8Altogether, 18 CBOs, each receiving 10 days of training, implemented the two interventions.

> 9Focus group discussions were held with sub-groups (young, women, etc.) in the community meetings. The intention was to let each group voice its concerns so as to reduce the risk of elite capture. Community score cards are used as a method aimed at both identifying and aggregating the issues to be addressed. In the community meeting, members scored the services provided by the facility on a scale from 0 to 100. In the health facility meeting, a similar exercise was implemented to identify key problems and constraints as viewed by the health staff, including a self-assessment of their performance. Role plays were primarily used in the interface meeting as a method to both illustrate and desensitize issues on which the provider and the community had differential views.

6
and the providers’ joint agreement on what needs be done to improve health care delivery, how, when, and by whom.

In total, the process of reaching an agreement took five days. After the meetings, the communities themselves had the responsibility to monitor the implementation of the issues outlined in the joint action plan. In 2008, about a year after the initial intervention, there was a one-day repeat engagement on a smaller scale facilitated by the CBOs during which health workers and community members discussed suggestions for sustaining or improving progress on the issues outlined in the joint action plan.

## **The participation & information intervention**

The _participation & information intervention_ mirrored the participation intervention with one important exception. Specifically, at the start of both the community and the health facility meetings, the facilitators provided the participants with quantitative data on the performance of the health provider. These data were collected from facility and household surveys implemented prior to the intervention. A unique report card was established for each facility summarizing information that was identified from the baseline data as key areas subject to improvement, including utilization, access, absenteeism, and patient-clinician interaction. The report cards also included comparisons vis-à-vis other health facilities and with the national standard for primary health care provision. The report cards were translated into the languages spoken in the community and posters were designed to help the non-literate process the provided information.

The _participation & information intervention_ was initiated at the beginning of 2005 and was followed by small scale repeat engagements in mid-2005, 2007 and 2008 as illustrated in figure 1. In between this fairly "minimal" intervention (a total of 10 days in 4 years), the communities were left to themselves to monitor the providers as outlined in the agreed upon action plan.

# **3.3 Data**

Data collection was governed by two objectives. The first objective was to create report cards for the _participation & information intervention_ on staff performance and how the community views the quality and efficacy of service delivery. The second objective was to rigorously evaluate the short- and long-run impacts.

To meet these objectives, two types of surveys were implemented: a survey of health care providers and a household survey in the catchment areas of the facilities. A quantitative service delivery survey was used to collect data from the providers. Because health providers may have a strong incentive to misreport key data, the data were obtained directly from the records kept by facilities for their own need (i.e., daily patient registers, stock cards, etc.) rather than from administrative records. The former, often available in a highly disaggregate format, were considered to 7
suffer the least from any incentive problems in record keeping. Data were also collected through visual checks by enumerators, including measuring absenteeism using unannounced visits.

Figure 3 provides details on the timing and scope of the data collection effort. The posttreatment survey collected data from 75 health facilities and roughly 7,500 households. While all surveys included a core set of modules, including socio-demographic characteristics, households’ health outcomes and health facility performance as experienced by the household in the household survey, additional modules were added in the later rounds. For example, the household survey in 2006 included modules on child mortality and anthropometric measurements (height of children aged below 5 and weight of infants). The household survey in 2009 also included an additional module on women’s health, including prenatal and antenatal care, and collected detailed birth and death statistics for children from 2004 and onward.

In addition to facility and household surveys, we have information from the action plans for the treatment group in the _participation & information intervention_ at the start of the intervention in 2005 and mid-way through when they were given the opportunity to revise their plans. We have similar data for the treatment group in the _participation intervention_ at the start of the intervention in mid-2007.

# **3.4 Outcomes and statistical framework**

We divide our empirical investigation into two parts. We start by analyzing the impact on the main outcome of interest; i.e., whether the intervention resulted in improved health outcomes. We have four sets of health measures: child mortality, pregnancy, birth and anthropometric measurements of children. Then, we turn to the quantity of health care. To measure utilization, we use data from the health facility (daily patient registers) as well as utilization data from the household survey.

The second part focuses on the channels through which the intervention may have influenced health outcomes and the demand and supply of health care. That is, we assess changes in all steps in the accountability chain; i.e. whether there is evidence of increased monitoring activities from the communities and information exchanges, using information derived from direct observations at the facility and data collected at the household level, and whether there is evidence that the health facility staff responded through improved treatment practices and overall management of the health clinics.

We use the same set of outcome measures as those used in the short-run evaluation of the _participation & information intervention_ (Björkman and Svensson, 2009). In addition, we report the findings on the extent to which clinical guidelines for the examination of patients were followed, with a focus on antenatal and postnatal care.<sup>10</sup>

> 10These data were only collected in 2009.

8
To assess the causal effect of the intervention, we compare mean outcomes, after accounting for stratification. That is, we estimate where _yijd_ is the outcome of household _i_ (when applicable), in community/health facility _j_ , in district _d_ , _Tjd_ is an indicator variable for assignment to treatment, _θd_ are district fixed effects, and _ε_ is an error term. _ijd_ For a subset of variables, we can also stack the pre- and post-data and estimate the following difference-in-difference specification where _POST_ is an indicator for the follow-up survey and _μj_ is a facility or community fixed effect. For some outcomes, we have a group of related outcome measures. To assess the impact of the intervention on a set ofunrelated regression system, _K_ related outcomes, we follow Kling et al. (2004) and estimate a seeminglyand derive average standardized treatment effects, _β_<sup> ̃</sup> = _K_<sup><u>1</u>∑</sup> _k_<sup>_K_</sup> =1 _σβ_ ˆˆ _<u>kk</u>_<sup>,</sup> where _β_<sup>ˆ</sup> _k_ is the point estimate on the treatment indicator in the _k_<sup>_th_</sup> outcome regression and _σ_ ˆ _k_ is the standard deviation of the control group for outcome _k_ (see Duflo et al. 2007).

# **4 Long-run impact of the participation & information intervention**

## **4.1 Balance at baseline**

Tables 1 and 2, panel A, report mean pre-treatment characteristics for the treatment and control groups in the _participation & information intervention_ and test statistics for equality of the means. We report both mean differences for a set of key outcome variables and differences in average standardized pre-treatment effects for each family of outcomes (utilization, utilization pattern, quality, catchment area statistics, health facility characteristics, citizen perceptions, supply of drugs, and user charges). There is no systematic difference between the treatment and the control group at the baseline. Thus, overall the sample is balanced.

9
# **4.2 Health outcomes and quantity of care**

## **Health outcomes**

The primary outcome measure for the trial was under-five mortality. We also consider four other health outcomes: number of births and pregnancies and height and weight of children. Child mortality links to the quality and the quantity of a wide spectrum of services that should be provided by dispensaries. Many of these services, including improved prenatal care, health education, child preventive care, and child curative care, also have the potential to affect height and weight. Height can be viewed as a summary measure of health and nutrition since conception, while weight is an indicator of current nutrition and illness status, especially for younger children. Number of births and pregnancies link directly to the quantity and the quality of family planning and health education.

Table 3 depicts the findings on child mortality for the period 2006-2009. We start by reporting the results using the raw data; i.e., the number of under-five, infant (under 12 months), and neonatal (under 1 month) deaths. The intervention reduced the number of deaths in all three age categories. The number of under-five deaths dropped by 32% (column i); the number of infant deaths dropped by 36% (column ii); and the number of neonatal deaths dropped by 51% (column iii).

The crude death numbers provide evidence of a fall in child mortality, but the reduction may not necessarily be due solely to a reduction in the risk of child death as cohort sizes may have been differentially affected by the intervention (for instance due to differential fertility rates). To account for this we estimate a mortality rate for children under-five (and infants) over the period of exposure; i.e., between January 2006 and May 2009. We follow the conventional approach used in epidemiology and define the mortality rate as the number of under-five (under 12 months) children that died during the period per per 1000 child-years of exposure over the same time period. We also, conventionally, define neonatal mortality as the number of neonatal deaths per 1000 live births. The results, with the data collapsed at the cluster level, are reported in columns iv-vi. We report treatment effects and for under-five and infant mortality also the rate ratio of the incidence of child deaths (occurrence of death over child-months) in the treatment versus the control group.<sup>11</sup> Column iv reports the under-five mortality effects. The estimated rate ratio; i.e., the ratio of the incidence of child deaths in the treatment relative the control group, implies a 23% reduced risk of under-five deaths in the treatment group. The effect is of the same order of magnitude, but less precisely estimated, using a linear model (a reduction of 4.43 deaths per 1000 child-years from a control group mean of 18.7 deaths per 1000). The reduction in infant mortality (column v) is slightly larger – a 28% reduced risk of infant deaths in the treatment versus the control group – but also less precisely estimated.<sup>12</sup> Column vi reports the treatment effect on neonatal mortality.

> 11The rate ratios are estimated using a Poisson regression model (Preston, 2005).

> 12Under-five and infant mortality rates can also be estimated using a Poisson model with individual (child level)

10
The number of children dying before reaching one month (per 1,000 live births) is 15.2 in the intervention group and 27.2 in the control group, which corresponds to a reduction in neonatal mortality of close to 44%.<sup>13</sup> As a reference point, the rate ratios for the short run impact of the _participation & information intervention_ on under-five and infant mortality are 0.74 and 0.68, respectively. That is, the reduction in child mortality in the longer run is similar to the short run findings, albeit somewhat smaller in magnitude.<sup>14</sup> Table 4 reports the findings on births and pregnancies. The intervention reduced the incidence (column i) and the number of births since 2006 (column ii) by 14% – an effect roughly twice as large in absolute values as compared to the short-run effect reported in Björkman and Svensson (2009). Columns (iii)-(iv) show that the effect is of the same order of magnitude – about two times larger in absolute values in the long (4 years) versus the short (1 year) run – when using the incidence and the number of pregnancies as dependent variables.

We measured the weight of children under-five using portable scales and the height of children under-five years using stadiometers. Table 5 reports the long-run intervention impacts on the two anthropometric outcomes. We report results separately for infants (0-11 months) and children (12-59 months) because reductions in illnesses tend to lead to gains in weight for infants while reductions in illnesses over time, and thus for older children, will tend to manifest in increased height. Columns (i-iii) depict the weight-for-age z-score results. Consistent with the findings in Cortinovis et al. (1997), and the short-run findings, Ugandan children have values of weight far lower than the NCHS/CDC international reference.<sup>15</sup> The treatment effect is 0.22 _z_ score in weight-for-age for infants and is fairly precisely estimated. Adding controls for age and gender does not change the results (column iii). There is no effect for children aged 12-59 months.

Columns (iv)-(vi) report the results for height of children, using height-for-age z-score as the dependent variable. To the extent that health care provision in the treatment group has experienced a sustained improvement, one could plausibly expect an impact on height. There is a positive treatment effect, significant at the 10 percent level, on height for children aged 12-59 months. Four years into the intervention, children aged 12-59 months in the treatment group are 0.10 z-scores taller than the children of the same age in the control group.

data. The point estimates from a model using individual level data are similar. The rate ratios [p-value] for under-five mortality and infant mortality are 0.77 [0.04] and 0.71 [0.07], respectively.

13The estimated neonatal mortality rate in the control group, 27.2, is similar to the neonatal mortality rate reported in official statistics; 25 in 2009. (World Bank Open Data)

> 14Björkman and Svensson (2009) did not collect data on the month of death. Thus, to derive these rate ratios we assume that children that died in 2005 died midway through the year. In Björkman and Svensson (2009) the under-five mortality rate is estimated by summing the death rates for each cohort (0-1 year old, 1-2 year olds etc.) per community. This life-table approach produces a probability of death rather than an incidence of child death as reported in table 3.

15Following Björkman and Svensson (2009), we omit observations with a recorded weight above the 90th percentile in the growth chart reported in Cortinovis et al. (1997). Since weight is measured by trained enumerators, the reporting error is likely due to misreported age of the child.

11
As illustrated in figure 5, and consistent with the finding of a positive treatment effect over the course of the program, the treatment effect is increasing in the exposure to treatment. For example, the treatment effect for children that have lived three years in a treatment area is 0.12 _z_ -score in height-for-age, corresponding to a 10 percent increase compared to the similar aged cohort in the control group.

# **Utilization**

Table 6 depicts the findings on utilization. Cross-section estimates based on equation (1) are given in Panel A. Estimates from a value-added specification, where we use average utilization for outpatient services and deliveries pre-intervention as a lagged variable, are shown in Panel B, and estimates from a difference-in-differences specification, equation (2), are reported in Panel C.<sup>16</sup> Four years after the intervention started, we find large positive effects on all four utilization measures. Although the point estimates are imprecisely estimated in the cross-section specification (see Panel A), the average standardized effect, reported in specification (v), is significantly different from zero. The precision improves and the point estimates are larger in the value-added specification, in which utilization for general outpatient services and deliveries are significantly different from zero at the 10 percent and 5 percent level, respectively. For both the value-added and the difference-in-differences specifications, the average standardized effects are highly significant.

The impact on utilization is substantial. For outpatient services, the increase goes from 16 percent in panel A to 27% in panel B. For deliveries, the point estimate in panel B suggests a 50 increase in utilization. For antenatal care, the increase varies between 21 percent (in panel A) to 25 percent in panel B.

The last three columns in table 7, Panel A and C, report changes in utilization patterns based on household data. We collected data on where each household member sought care in case of illness that required treatment. As is evident, households in the treatment communities switched from traditional healers and self-treatment, specification (vii), to the project facility, specification (vi), in response to the intervention. The average standardized treatment effects, reported in specifications (viii) and (xix), are significant in both the cross-section and the panel model.

> 16We have pre-intervention data for outpatients and deliveries but not antenatal care and family planning. Thus, we use the average utilization for outpatient services and deliveries pre-intervention as a lagged variable in the value-added model and estimate

_yjst_ = _αVA_ + _βVATj_ + _λ_ ̄ _yjt_ −1 + _εj_ .

where _ysjt_ is utilization for service _s_ in facility _j_ at time _t_ , and _y_ ̄ _jt_ −1 is the average utilization for outpatient services and deliveries in the pre-intervention period _t_ − 1. The difference-in-differences models can only be estimated on outpatient services and deliveries for which we have pre-intervention data.

12
# **4.3 Processes and health treatment practices**

The findings in section 4.2 show that the intervention, which resulted in large improvements in health outcomes and utilization in the short run (Björkman and Svensson, 2009), also resulted in long-run improvements in health. Below we try to identify possible channels through which these changes came about. The intervention was intended to improve health outcomes by strengthening the community’s ability to monitor and hold providers to account which, in turn, would influence health workers’ incentives to provide both more and better health care. We next turn to assessing the evidence for these two links.

## **Processes**

In table 7, we use data on a set of process measures that proxy for the community’s ability and extent of monitoring and engagement. The data in panel A were collected through visual checks by the enumerators during the post-intervention health facility survey. Four years after the initial intervention, the average standardized effect of having a suggestion box, numbered waiting cards, a staff duty roaster, and posters informing patients about their rights and that services are free is significantly positive. For two of the variables, having a suggestion box and a staff duty roaster, the individual effects are also significantly positive and the point estimates, relative to the control group, are fairly large.

Panel B uses information collected at the household level. There are positive long-run effect on all individual indicators; i.e. whether the household has received information about the HUMC’s role and responsibilities, whether the performance of health workers was discussed in Local Council meetings, whether a household member told the staff about the household’s (dis)satisfaction with the health services received, whether they think that the staff at the facility works closely with the community, and whether they know somebody who monitored the health facility and staff performance.

## **Health treatment practices**

Tables 8-11 focus on health workers’ behavior; i.e. treatment practices and management.

Table 8 looks at a set of generic measures of behavior. The intervention had no significant long-run impact on the likelihood that any equipment was used during the patient’s last examination, column (i), or waiting time before being examined, column (ii). Note, however, that as utilization increased in the treatment group (see table 6), the total number of patients examined with equipment increased. Column (iii) reports the impact on absenteeism based on data from three unannounced visits. The point estimate is close to zero and insignificant. There is suggestive evidence, however, that the absence rate for staff residing in the community fell in the treatment clinics (the results are available upon request).

The general condition of the clinic – a measure of the conditions of the floor, walls and furniture 13
and the smell of the clinic – remained significantly better in the treatment group also in the long run (column iv).<sup>17</sup> The effect is substantial, with the mean treatment facility having a 0.5 standard deviation higher score compared to the average control facility. There is also a significant decrease in the probability of drug stock-outs in the long run, column (v), although there is no systematic difference in the supply of drugs between treatment and control groups.<sup>18</sup> Table 9 focuses on the extent to which clinical guidelines for the examination of patients were followed. We collected detailed data on antenatal and postnatal care. Four years after the initial intervention, health workers in the treatment communities appear to provide more effort, or better quality, in their interactions with pregnant women and infants. During antenatal care visits, pregnant women were more likely to be examined by a midwife; to have their weight taken; to have a blood sample taken; to have the fetus checked; and to receive information about pregnancy complications. Newborn children were also more likely to be checked at the facility in the first two months after delivery. For clinical guidelines that are seldom followed in the control group – checking the newborn in the first two months after delivery (a 24 percent increase in the treatment compared to the control group) and measuring weight and taking blood samples of pregnant women (a 22 percent increase in the treatment compared to the control group) – the treatment effects are substantial.

Table 10 includes results on health education on three of the most prevalent diseases in Uganda: HIV/AIDS, tuberculosis and malaria. Column (i) shows, not surprisingly given the mean in the control group of 0.99, that there is no impact on whether household members had heard of AIDS. But the intervention decreased the probability that the household members expressed stigmatization about AIDS (as measured by agreeing with the statement that people living with AIDS should be ashamed of themselves), increased the knowledge that tuberculosis is spread through the air, and increased the probability that children were sleeping under a treated mosquito net.

Table 11 reports the impact on immunization by children’s age group.<sup>19</sup> Two results stand out from the table. First, while the point estimates are positive, the treatment effects are small (between 3%-6%) and insignificantly different from zero. Second, coverage in the control group varies between 78%-94%, which is a doubling of the immunization rates since 2005. As almost nine out of ten children received immunizations according to the recommended immunization

> 17The enumerators visually checked the condition of the health clinics; i.e. whether floors and walls were clean, the condition of the furniture and the smell of the facility. From these data, we constructed a summary indicator using a principal components analysis. The variable general condition of the clinic is the standardized (using control group mean and standard deviation) first component.

> 18See section 6.3.

> 19For each age group, we use information on how many times (doses) in total each child has received of polio, DPT, BCG, and measles vaccines and vitamin A supplements. On the basis of the recommended immunization plan, we create indicator variables taking the value of 1 if child i of cohort (age) _j_ had received the required dose(s) of measles, DPT, BCG, and polio vaccines, respectively, and 0 otherwise. We then estimate equation (3), for each age group, and calculate average standardized effects.

14
plan, there is little room for improvement. Moreover, the sample sizes in table 11 are insufficient to detect small significant treatment effects at standard levels of significance.<sup>20</sup>

# **5 Impact of the participation intervention**

The short-run evaluation of the _participation & information_ intervention showed that efforts to stimulate community monitoring can have large effects on both the quantity and the quality of service provision and in the end result in improved health outcomes (Björkman and Svensson, 2009). The results presented in Section 4 show that the main results were sustained four years into the intervention. Collecting, assembling, and disseminating information on performance in a rural setting, however, is both technically complicated and costly. Moreover, as the intervention involved both information dissemination and various techniques to stimulate engagement and participation, it is unclear if and to what extent both components are crucial in order to initiate a successful process of community involvement and monitoring.

To address these issues, we designed a new intervention in 2007. The _participation intervention_ was designed so as to mimic the _participation & information_ intervention in all aspects but one – the participants in the community and health facility meetings did not receive quantitative information on the performance of the provider. Hence, the communities’ informational constraints were not addressed in the _participation intervention_ .

## **5.1 Balance at baseline**

As is evident from panel B in table 1, columns (iv)-(vi) and panel B in table 2, column (ii), there are no systematic differences between the treatment and the control group as concerns any of the baseline characteristics. Thus, the sample is balanced. Comparing characteristics of the control groups in the two experiments in the same year (2006) also shows that there was no significant difference across the comparison groups in the two experiments (see Table A.1 for details).

## **5.2 Main outcomes: Health outcomes and quantity of care**

### **Health outcomes**

> 20The World Health Organization and other UN agencies have worked intensively with the Ministry of Health in Uganda on large-scale and country-wide immunization campaigns during the period of the study. This may explain the improvement in immunization coverage in both the treatment and the control group.

15
Table 12 reports the short-run findings on health outcomes. The control group means are similar to those reported in tables 3-5 for the _participation & information_ intervention, but the treatment effects are small and insignificant.<sup>21</sup> That is, there are no statistical differences in under-5 mortality (column i); infant mortality (column ii); neonatal mortality (column iii); number of births (column iv); number of pregnancies (column v); and in weight-for-age and height-for age z-scores (columns vi-ix) between the treatment and the control group. Thus, based on health outcomes only, the participation intervention had no impact.

### **Utilization**

Table 13 reports the results on utilization. There is no clear evidence of impact. For general outpatient service, the point estimate goes from –100.2 (p-value=0.38) to –20.3 (p-value=0.89) depending on the specification. For delivery, there is a significant negative effect in two out of three specifications, while the treatment effects are positive, albeit insignificant, for antenatal care and family planning. The average standardized treatment effects are small and insignificantly different from zero in all three specifications.

Columns (vi) and (vii) report changes in utilization patterns based on household data. There is some evidence that treatment households are less likely to use traditional healers and selftreatment, but the average standardized treatment effects are once more insignificant.

## **5.3 Processes and health treatment practices**

The _participation intervention_ did not result in improvements in child health or the quantity of health care used. In the appendix, tables A.2-A.6, using the same outcome variables as in section 4.3, we show that there is no systematic evidence that the intervention resulted in increased monitoring and information exchange in the treatment communities. Management of the facility and the extent to which clinical guidelines were followed also remained unchanged.

# **6 Mechanism and Discussion: The role of information**

The results of the _participation_ and the _participation & information_ interventions differ markedly. Without information, the process of stimulating participation and engagement had little impact on health workers’ performance or the quality of health care. In contrast, when community members

> 21The p-values for the null hypothesis of equal means in the two control groups are 0.65 (for under-5 mortality); 0.62 (for infant mortality); 0.74 (for neonatal mortality); 0.22 (for number of births); 0.28 (for number of pregnancies); 0.50 (for weight-for-age); and 0.17 (for height-for age).

16
are informed about staff performance, the same type of process intervention resulted in significant improvements in health care delivery and health outcomes in both the short and the longer run.

In this section, we investigate why the provision of information appeared to have played such a crucial role.<sup>22</sup> We start by presenting a stylized example of why information may matter in these kinds of programs. Then, we use process data from the two treatment groups to identify differences in decisions and actions taken in the two treatment groups (with informed and uninformed community members). We discuss alternative mechanisms and also review the literature on beneficiary control and argue that a key difference between successful and unsuccessful community empowerment interventions is access to information on the staff’s behavior.

# **6.1 What role does provision of information play?**

Why is access to information about the staff’s performance a constraint to the community’s ability to monitor and hold providers to account? Data from the baseline survey provide some suggestive evidence. For a set of indicators, like waiting time and absenteeism, we have data both from the in-charge of the facility and data collected either through visual inspections (by enumerators) or collected directly from the users. The comparison reveals a systematic pattern: the in-charge underreport performance problems. For example, the mean waiting time as reported by the clinician was 2 minutes. The household survey data, however, revealed that it was close to 2 hours. Data from unannounced staff surveys showed an absence rate of around 50 percent. In the in-charge survey, however, the clinic officers did not report problems with absenteeism and explained the low presence at the time of the survey as the result of a high, but unsubstantiated, level of staff training and outreach. Moreover, when the in-charge was asked to list key constraints facing the clinic, lack of funding, staff, material, and drugs were high on the agenda, while issues such as weak adherence to clinic guidelines, shirking, or mistreatment of patients were never listed.

Motivated by these examples, we provide a simple stylized model of how the provision of information on mean outcomes, by enabling users to better distinguish between health workers’ effort and factors outside the health workers’ control, can influence users’ willingness to take actions (to monitor the provider) and, in the end, health workers’ performance.

Consider a setup where the quality of health service at each visit _j_ and time _t_ , _qjt_ , depends on two factors that are unobserved by the patient: the health worker’s performance ( _pt_ ) and resources provided from the center ( _rjt_ ), where

> 22Lack of information was highlighted as a key constraint in the earlier work on community empowerment (see Samuel Paul 1987; Jenkins and Goetz 1999; and Goetz and Jenkins 2001).

17
Assume that the resource flow from the center, _rjt_ , is a random variable taking the value of 1 with probability _μ_ and 0 with probability 1 − _μ_ . The health worker’s performance is a function of the health worker’s effort ( _et_ ). With probability _e_ , the performance is "appropriate" or "high" ( _p_ = 1) and with probability 1 − _e_ , the performance is "inappropriate" or "low" ( _p_ = 0). The health worker chooses an effort which costs him _c_ ( _e_ ) = 2<sup><u>1</u></sup><sup>_e_2.</sup> There are two types ( _T_ ) of health workers: an honest type ( _H_ ), who always chooses _e_ = 1, and a opportunistic type ( _O_ ), who chooses _e_ so as to maximize his expected utility. Nature chooses type at the start of the period with a worker of type _O_ chosen with probability _φ_ = 1/2. As an honest type always chooses high effort, our focus is on the opportunistic type.

Patients do not observe _pt_ , _et_ , or _rjt_ . However, they know the distribution of _rj_ and _φ_ and observe _qjt_ after each visit. Patients seek service at the health center and care about the (expected) quality of health service _qjt_ and the cost of taking local actions. As our focus is on the role of information, we disregard collective action problems and can then drop subscript _j_ and just assume that there is one representative community member (patient). We treat the representative patient as myopic, in the sense that she does not aggregate information over visits. Thus, the representative patient observes _qt_ = 1 in period _t_ with probability _eμ_ and _q_ = 0 with probability 1 − _eμ_ . Based on the outcome, the patient can take local actions at a cost _κ_ , which inflicts a cost _λ_ (1 − _e_ ) on the provider, where _λ_ ∈ (0, 1). We can think of the costs _λ_ (1 − _e_ ) as capturing the shame a low performing health worker would suffer if the community expressed its displeasure with his performance. The costs are proportional to (1 − _e_ ) as the cost inflicted on the health workers is increasing in the extent of shirking.

An opportunistic health worker’s (per-period) payoff _π_ is where _w_ is the wage rate and _L_ is a binary variable taking the value of 1 if the community decides to take actions against the provider.

The sequence of events is as follows. Nature chooses type at the start of period 1. The health worker then makes an unobserved choice of effort ( _e_ 1) and the resource flow from the center ( _r_ ) is realized. The (representative) patient experiences the quality of health service, _q_ 1, and updates her beliefs about the type of health worker. Depending on the outcome, the users then make a decision of whether or not to take local action and the health worker makes a new effort choice ( _e_ 2). Finally the patient experiences the quality of health service in period 2 ( _q_ 2). We denote by _φ_ ( _φ_ | _q_ 1, ̃ _e_<sup>_T_</sup> ) or _φq_ the posterior probability that the provider is an opportunistic type, given a realized outcome _q_ 1, a prior probability _φ_ , and the user’s expectation of the health worker’s choice of effort conditional on type _T_ ( _e_ ̃<sup>_T_</sup> ).

18
_Assumption 1: μ_ > _κ_ / _λ_ > _μ_ /(2 − _μ_ ) _Scenario 1_ : Suppose that no information is provided. Then, provided that assumption 1 holds, it is optimal for the community not to take any local actions. The (opportunistic) health worker exerts no effort ( _e_ 1 = _e_ 2 = 0) and the quality of health service is low ( _q_ 1 = _q_ 2 = 0).<sup>23</sup> _Scenario 2_ : Suppose that patients receive information on the mean quality of health service; i.e. _Eq_ [ _q_ ] = _q_ ̄. As expected quality conditional on the worker being an honest type is _Eq_ [ _q_ | _H_ ] = _μ_ and an opportunistic type at most exerts effort _e_ ≤ _λ_ < 1, implying that _Eq_ [ _q_ | _O_ ] ≤ _λμ_ , patients can tell with certainty, given _q_ ̄, whether the worker is an honest or an opportunistic type. If assumption 1 holds, the community will therefore take local actions and the health worker will respond by exerting a higher effort ( _e_ 2 = _λ_ ). The expected period 2 quality is _λμ_ .

_Intuition_ : Provision of information solves patients’ aggregation problem. This enables users to distinguish between health workers’ type or effort and factors outside the health workers’ control. As a result, the return to local action increases. In equilibrium, community members become more involved; i.e., take local actions, and the health worker responds by exerting higher effort. Note that assumption 1 does not hold for _μ_ close to 0 and 1. In the first case, public service provision is so poor that the health worker’s choice of effort has almost no bearing on health quality, so there are low returns to incentivize the provider. In the second case, the community member can tell with certainty, if observing _q_ = 0, that the health worker is an opportunistic type so that there is no aggregation problem. Moreover, the difference between the two thresholds, _μ_ − _μ_ /(2 − _μ_ ), has an inverse U-shape with a maximum at _μ_ = 2 − √2; that is, provision of information is most likely (i.e. the parameter space for which assumption 1 holds is the largest) to have an effect when public support to frontline providers is neither too effective or too poor.

# **6.2 Information and local actions**

The above stylized example shows how the provision of information on mean outcomes, by enabling users to better distinguish between health workers’ effort and factors outside the health workers’ control, can influence the community members’ willingness to take actions. In table 14 we use data from the implementation phases of the two interventions to corroborate this prediction.

A key component of both experiments was the agreement of an action plan. The action plan outlined the community’s and the provider’s joint agreement on what needs be done and by whom. As shown in table 14, the process of reaching an agreement looks similar on some observable characteristics. About the same number of community members participated in the community meetings (column i). The two treatment groups also identified roughly the same number of actions

> 23See the appendix for details.

19
to be addressed (column ii). A closer look at the type of actions agreed upon, however, reveals that while the _participation group_ mostly identified issues that required third-party actions; e.g. more financial and in-kind support from upper-level authorities and NGOs and a timely delivery of medicines from the center, the _participation & information group_ almost exclusively identified (88 percent on average) local issues, which either the health workers or the users could address themselves, including absenteeism, opening hours, waiting time, and patient-clinician interactions (columns iii and iv). Even two years into the _participation & information intervention_ , i.e. in 2007, when a shorter repeat engagement was implemented, more than four-fifths of the actions identified in the joint action plans dealt with local issues (column v). Figure 6 illustrates the main result.

The difference in the content of the action plans across the two treatment arms and the difference in the impact (health outcomes and staff behavior) of the two interventions are consistent with the hypothesis that when the principal (i.e. community) is informed, efforts to stimulate beneficiary control result in an actionable plan that affects staff behavior and thereby health utilization and health outcomes. That is, the report card component helped build a reform agenda on the true as opposed to the perceived status of service provision. But the findings do not rule out other explanations. To further examine the plausibility of the information and local actions channel as a key mechanism for the health utilization and health outcomes treatment effects, we exploit variation within the two treatment arms. Specifically, we use the difference in the number of outpatients served before and after treatment in treatment clinics as compared to the mean difference in the ̄ = − ̄ _C_ ̄ control clinics as the dependent variable; i.e., _yj_ � _ytj_<sup>_T_−</sup><sup>_y_</sup> _t_<sup>_T_</sup> −1 _j_ � � _yt_<sup>−</sup><sup>_y_ ̄</sup><sup>_C_</sup> _t_ −1�, and regress _yj_ on the share of local issues raised in the action plan. The estimated relationship between the difference in the number of outpatients served before and after treatment compared to control clinics, conditional on the share of local actions agreed upon in the action plan, is illustrated in figure 7. There is a consistent pattern across clinics and treatment arms. The _participation & information group_ identified a significantly larger share of local actions and a higher share of local actions is associated with a larger, and significant, increase in the number of outpatients served over time and as compared to the control group.

# **6.3 Alternative mechanisms**

The findings of large treatment effects in the _participation & information_ intervention are consistent with the community-based monitoring mechanism discussed above. But the findings do not rule out other explanations. In particular, other agents in the supply chain, e.g., district or subdistrict management, may have changed their behavior or support in response to the intervention, for example by providing additional funding or other support to the treatment facilities. We do not find any evidence of this being the case (see the appendix, table A.7). The treatment facilities did 20
not receive more drugs or funding from the sub-district or district as compared to the control facilities. Moreover, the level of supervision of providers by government elected authorities remained low in both the treatment and the control group. There is also no difference between treatment and control facilities in the number of staff that voluntarily left the facility.

# **6.4 Discussion**

Over the past decade, the World Bank alone has allocated close to $85 billion to local participatory development programs. A comprehensive review of those programs (Mansuri and Rao, 2013) notes that few programs are systematically evaluated and that the enthusiasm for participatory initiatives is driven more by ideology and optimism than by analysis, either theoretical or empirical.

In this paper, we present the results of two field experiments. First, we assess the longer run impact of an intervention combining the standard package of facilitated meetings to enhance participation with the dissemination of report card information on the facility’s performance. While programs to stimulate community involvement are nowadays common in most developing countries, there is little evidence of whether and when these types of bottom-up initiatives have the intended impact and no evidence based on rigorous evaluations of the longer run impact.

Second, we assess the short-run impact of an intervention involving only the standard package of facilitated meetings to enhance participation. The focus on participation alone is typical for most Community Driven Development (CDD) programs.

The process of stimulating participation, when the community does _not_ have access to information on performance, resulted in a joint agreement focusing on issues requiring third-party actions. The intervention had little impact on health workers’ behavior or the quality of health care. In contrast, with an informed community, the same type of process intervention resulted in a joint agreement almost exclusively identifying issues that either health workers or the users could address themselves, and significant improvements in health care delivery and health outcomes in both the short and longer run.

The results of the two field experiments resonate well with the mixed findings in the literature. Banerjee et al. (2010) study beneficiary control in Uttar Pradesh, India. In one of their treatment arms, trained facilitators held small-group discussions with parents and provided them with simple tools to enable them to generate their own information about their children’s learning outcomes. This information was later shared in a village meeting where the school teachers were also asked to provided general information about the resources available at the school. No information was provided about the teachers’ performance. While test scores are likely to be correlated with the teachers’ effort, it is plausible that parents would (correctly) infer that the variation in test scores is to a large extent driven by child- and household-specific factors, rather than teacher effort, and 21
they might, possibly incorrectly, infer that the lack of resources that has been highlighted in the meetings is also a quantitatively important factor. Thus, even if parents viewed these test scores as a credible and objective measure of what their children know, they might be skeptical about using these test scores as an indicator of the school’s performance. As in the _participation intervention_ evaluated above, the village meetings were dominated by discussions about third-party actions (e.g. the state government’s scholarship program and a new school-meals program), not the teachers’ performance or the learning environment. The intervention prompted no increased teacher effort and no improvement in educational outcomes.

Olken (2007) evaluates different ways of monitoring corruption in a road construction project in Indonesia. In one of the experiments, invitations were sent out to village-level meetings where project officials documented how they spent project funds for local road construction. This provided villagers with new information, but at best only indirect information about the key outcome variable – corruption – as project officials and/or elite community members may be able to hide it when reporting on how funds were used.<sup>24</sup> Thus, it is unclear whether the non-elite community members were really more informed about corruption in the project and consistent with the above results, corruption problems were seldom discussed in the village meetings and the intervention had little impact.

Duflo et al. (2012) evaluate a contract teachers program in Kenya. One of their treatment arms included School-Based Management (SBM) training of the PTA committee. Committee members were then given specific training on how to monitor and assess teachers’ effort and performance and a set of parents were asked to perform attendance checks on the teachers on a regular basis. A formal sub-committee of parents was formed to evaluate the contract teacher and deliver a performance report at the end of the year. While this intervention did not directly provide quantitative information on performance, it provided detailed training to the community on how to measure it throughout the year and how to combine the information in a performance report that could be shared with others.<sup>25</sup> SBM training for PTA committees reduced teachers’ absenteeism and increased student test scores.

Our findings provide both encouraging, and less encouraging, news for those promoting greater beneficiary control. On the one hand, we show that a standard, and CDD inspired, beneficiary involvement intervention, where the core of the strategy is the process through which (local) decisions are made, had no impact. On the other hand, we show that the same process based intervention in a scenario where the community is informed about how the public health facility is performing resulted in large and long-run improvements in both health service provision and health

> 24Corruption is not easily observable and measurable, as evidenced by Olken’s (2007) novel but burdensome strategy to estimate it.

> 25In addition, the intervention provided the PTA with hard incentives (the right to fire or retain the contract teacher).

22
# outcomes.

Collecting data on performance using traditional survey methods is a costly and to some extent technically complex endeavor. Thus, although our findings point to large and sustained positive health effects, it is unclear whether such an intervention is possible to scale up. This opens up important questions for future research. For example, is it possible to provide beneficiaries with tools to collect performance data and strategies to use them also in the health sector, for instance along the line of Duflo et al. (2012), instead of directly providing them with report cards? Can recent advances in ICT be used to identify ways of assembling and disseminating information at a lower cost?

23