metadata: authors: 0: author: Shapira,Gil
1: author: Kalisa,Ina
2: author: Condo,Jeanine
3: author: Humuza,James
4: author: Mugeni,Cathy
5: author: Walldorf,Jeanette
count: Rwanda
docna: 0: docna: The effects of in-kind demand-side
            conditional transfers for improving uptake of maternal and
            child health services in Rwanda
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Reproductive Health,Early Child and Children's Health,Health Service Management and Delivery
teratopic: Health, Nutrition and Population,Poverty Reduction,Macroeconomics and Economic Growth,Social Protections and Labor
abstracts: cdata!: To diagnose and treat preventable
            threats to maternal and neonatal health in Sub-Saharan
            Africa, a policy focus has been put on increasing coverage
            rates of targeted health services. Exploiting an
            experimental design, this study evaluates the impacts of an
            in-kind conditional transfer intervention in Rwanda that
            endowed women with gifts for receiving timely antenatal and
            postnatal care, as well as for delivering in health
            facilities. The analysis finds that although health centers
            experienced frequent stock outs of the gifts, the rate of
            women who initiated antenatal care within the first four
            months of their pregnancy increased by 7.7 percent, and that
            of women who received postnatal care in the 10 days
            following delivery increased by 8.6 percent. No impact was
            found on the rate of in-facility deliveries, which
            independently sharply increased during the years of the
            implementation of the program.
geo_reg_key: 517196,517191
repnme: repnme: The effects of in-kind demand-side
            conditional transfers for improving uptake of maternal and
            child health services in Rwanda
display_title: The effects of in-kind demand-side
            conditional transfers for improving uptake of maternal and
            child health services in Rwanda
listing_relative_url: /research/2017/05/27470932/effects-in-kind-demand-side-conditional-transfers-improving-uptake-maternal-child-health-services-rwanda
topicv3: Infant and Child Health m1326775 642,Sexual Reproductive, Maternal and Child
            Health m1327262 639,Antenatal and Postnatal Care m1327264 1451,Health Services Delivery m1327294 1305,Health m1504661 3480
publishtoextweb_dt: 2017-05-16T00:00:00Z
historic_topic: Health, Nutrition and Population,Poverty Reduction,Macroeconomics and Economic Growth,Social Protections and Labor
no_of_pages: 21
author_keys: 0: author_key: 964001
1: author_key: 1068365
2: author_key: 1294776
model_extractions: WPS8060 Policy Research Working Paper 8060 The Effects of In-Kind Demand-Side Conditional Transfers for Improving Uptake of Maternal and Child Health Services in Rwanda _Gil Shapira Ina Kalisa Jeanine Condo James Humuza Cathy Mugeni Jeanette Walldorf_ Development Research Group Human Development and Public Services Team May 2017
Policy Research Working Paper 8060

# **Abstract**

To diagnose and treat preventable threats to maternal and neonatal health in Sub-Saharan Africa, a policy focus has been put on increasing coverage rates of targeted health services. Exploiting an experimental design, this study evaluates the impacts of an in-kind conditional transfer intervention in Rwanda that endowed women with gifts for receiving timely antenatal and postnatal care, as well as for delivering in health facilities. The analysis finds that although health centers experienced frequent stock outs of the gifts, the rate of women who initiated antenatal care within the first four months of their pregnancy increased by 7.7 percent, and that of women who received postnatal care in the 10 days following delivery increased by 8.6 percent. No impact was found on the rate of in-facility deliveries, which independently sharply increased during the years of the implementation of the program.

This paper is a product of the Human Development and Public Services Team, Development Research Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at gshapira@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **The Effects of In-Kind Demand-Side Conditional Transfers for Improving Uptake of Maternal and Child Health Services in Rwanda**

Gil Shapira<sup>a</sup> , Ina Kalisa<sup>b</sup> , Jeanine Condo<sup>c</sup> , James Humuza<sup>b</sup> , Cathy Mugeni<sup>d</sup> , Jeanette Walldorf<sup>a</sup> <mark>JEL classification codes: I12, O15, H51</mark> ____________________________________________________________________________________ a Development Research Group, The World Bank b School of Public Health, College of Medicine and Health Sciences, University of Rwanda c Rwanda Biomedical Center d Ministry of Health, Rwanda

* Corresponding author: Gil Shapira, gshapira@worldbank.org

The Community PBF program was implemented by the Ministry of Health, under the leadership of Dr. Agnes Binagwaho, Fidele Ngabo and Cathy Mugeni. The authors would like to acknowledge the contribution to the program and its evaluation made by Netsanet Walelign Workie, Jennifer Sturdy, Gayle Martin, Sebastian Martinez, Paul Gertler, Paulin Basinga, Patrick Mullen, Christel Vermeersch, Elisa Rothenbuhler, Vedaste Ndahindwa and Randy Wilson. The team benefited from comments made by participants of seminars at Erasmus University Rotterdam and Universitat Pompeu Fabra. The financial contributions of the Health Results Innovation Trust Fund are gratefully acknowledged. The findings, interpretations and conclusions expressed in this paper are entirely those of the author, and do not necessarily represent the views of the World Bank, its Executive Directors, or the governments of the countries they represent.
# Introduction

Maternal and child mortality rates have reduced significantly in Sub-Saharan Africa between 1990 and 2015. It is estimated that the under-five mortality rate reduced by 52 percent and maternal mortality rate by 49 percent (United Nations 2015). However, both mortality rates in the region remain about twice as high as the corresponding world averages, and the ambitious targets set by the Millennium Development Goals have not been achieved. Further reducing these rates has remained in the heart of the post-2015 international development agenda and new targets were set to be achieved by 2030 by the Sustainable Development Goals.<sup>1</sup> In an attempt to further improve maternal and child survival, a focus has been put on increasing coverage rates of antenatal care, skilled-attended delivery and postnatal care of both mothers and children.

This study evaluates the impact of an intervention that incentivized users of health services with in-kind conditional transfers. In particular, the intervention endowed women with gifts for receiving timely antenatal and postnatal care, as well as for delivering in health facilities. The objective of the demandside incentives strategy was to increase use of these targeted services in order to diagnose and treat preventable threats to maternal and neonatal health. The intervention was introduced in the context of the Community Performance-Based Financing (CPBF) program, an extension to the national Performance-Based Financing (PBF) program at the health facility level.

The evaluation relies on an experimental design in which funding for the implementation of the demand-side incentives was given to health centers in randomly selected sectors (sub-districts). Facilities received this funding for a period of about two and a half years. The analysis relies on a baseline household survey conducted prior to the launch of the program and a follow-up survey of households and health centers conducted about nine months after last funds were transferred to the facilities.

The results show that demand-side in-kind incentives can effectively increase utilization of targeted services. They caused an increase of 7.7 percentage points in the rate of women who initiated antenatal care within the first 4 months of their pregnancy and an increase of 8.6 percentage points in the rate of women who received postnatal care in the ten days following delivery. No significant impact was found on the rate of women who delivered in a health facility attended by a skilled health provider. However, the rate of in-facility skilled-assisted deliveries sharply increased across the board and reached 95 percent at the time of the follow-up survey.

These results are consistent with previous studies of schemes that conditioned demand-side transfers on health service utilization.<sup>2</sup> Banerjee, Duflo et al. (2010) report results of a clustered randomized controlled study in rural India in which provision of lentils and metal plates to families whose children got immunized significantly increased rates of full immunization. They conclude that the small incentives had a large impact and were more cost effective than improving the reliability of the availability of the immunization services. Other studies have evaluated the impacts of financial incentives. For example, Thornton (2008) finds that even small monetary incentives can double the share of individuals who learn results of HIV tests in Malawi. A cluster of studies evaluate the impacts of multi-sectoral social

> 1 www.un.org/sustainabledevelopment 2 Lagarde et al. (2008) and Gopalan et al. (2014) provide relevant reviews of the literature.

2
protection programs in Latin America that included incentives for preventative health services (e.g. Gertler and Boyce 2001; Fernald Gertler and Neufeld 2008; Barham 2011; Barham and Maluccio 2011). Overall, these programs are found to be effective in increasing coverage of preventative health services and some studies even found positive impacts on the health status of beneficiaries.

This study contributes to this literature by evaluating a large-scale government-implemented program in Sub-Saharan Africa that endowed conditional in-kind transfers. Unlike the programs in Latin America, the policy intervention was not part of a bigger social protection program and was primarily implemented by health centers that procured and distributed the gifts. Unlike the experiments studied by Thornton (2008), which also took place in Sub-Saharan Africa, and Banerjee, Duflo et al. (2010), which also provided in-kind transfers, the CPBF scheme is a large-scale multi-year program. Evaluating the impacts of a program run by the government on a large scale, rather than by a research team or an NGO, is of value by itself as it might be implemented differently. For example, health centers experienced frequent stock outs of the gifts. That might have not occurred under a more controlled small-scale implementation of such a program.

# Background and the Intervention

Since the devastating 1994 genocide, Rwanda has experienced impressive improvements in health outcomes and utilization rates of health services. According to the Rwanda Demographic and Health Surveys, the infant mortality rate has decreased from 107 to 32 per 1,000 live births between 2000 and 2015 (NISR, MOH and ICF International 2012; 2015). Under-five mortality has decreased from 196 to 50 per 1,000 live births during the same period. The increase of in the share of births occurring in health facilities from 27 percent to 91 percent illustrates the increase in utilization of health services.

During the past two decades, the health sector in the country went through major reforms such as decentralization and the scaling up of the community-based health insurance program. One of the policy interventions that has been shown to contribute to improvements in maternal and child health indicators is the national Performance-Based Financing (PBF) scheme. In this pay-for-performance scheme, health facilities receive financial incentives for the use and quality of selected health services. evaluations of the program have concluded that it has significantly increased the health outcomes of children, quality of prenatal care and coverage of health services such as institutional deliveries, preventative care visits by children and voluntary counseling and testing for HIV by couples (Basinga, Gertler et al. 2012, Gertler and Vermeersch 2012 and de Walque, Gertler et al. 2015).

The CPBF program was designed with the objective of further expanding coverage of maternal and child health services and improving the quality of monitoring data collected at the village level. The program tries to achieve these objectives through expansion of performance-based payments to additional agents in the health system. A focus has been put on services whose coverage was not impacted by the PBF at the health facility level. The primary component of the program is performance payments to cooperatives of community health workers. The payments were given for timely completion and submission of reports on health indicators at the community level and for utilization of targeted health services by the communities the cooperatives serve.

3
This study evaluates the impacts of another component of the program, demand-side incentives. Health centers received funding to endow women with gifts for meeting the following eligibility criteria: initiation of antenatal care within the first four months of a pregnancy, delivery in a health facility, and initiation of postnatal care within ten days of delivery.<sup>3</sup> Table 1 summarizes the eligibility criteria and the monetary values set as ceilings for each service. Gifts of a value of up to 5 USD were to be endowed for timely antenatal care and values of 6.67 and 3.33 USD were set for in-facility delivery and timely postnatal care respectively.<sup>4</sup> The facilities were provided with suggested content for the incentive packages. Women who received these in-kind transfers during a pregnancy were not eligible to receive them again for a duration of three years.

The program launched in October 2010 and facilities received last funding for the gifts in February 2013. When the program was launched, procurement of the gifts was done centrally by the Ministry of Health but procurement responsibility was shifted to the individual health centers. There were frequent reports of stock-outs of the gifts and many women who fulfilled eligibility criteria did not receive the gifts.

# Experimental Setup and Data

The demand-side in-kind transfers program was randomly introduced at the sector level.<sup>5</sup> For the impact evaluation of the CPBF, 19 districts were selected in four provinces (excluding the province of the capital city Kigali). These were the 18 districts participating in the impact evaluation of the PBF program in 2006-2008 and an additional district that was added to increase the statistical power of the study. The demand-side intervention was piloted in 30 sectors, and those were excluded from the study sample. Additional sectors were excluded if they did not contain a public or non-for-profit faith-based health center.<sup>6</sup> Sectors were blocked by district and poverty ranking, as defined by a social protection program (Vision Umurenge Program), and then randomized into 4 study arms.

In all study-arms, community health worker cooperatives were paid conditional on the completion of quarterly reports. In two of the study arms, cooperatives were also paid for the utilization of targeted services (pay-for indicators). Two study arms were selected for the implementation of the demand-side incentives, one with pay-for-indicators and one without. Forty-eight sectors were assigned to the payfor-reports intervention only and 50 sectors were assigned to each of the three other groups. This analysis focuses only on the impact of the demand-side transfer and compares the two study arms that implemented this program with the two that did not. Therefore, for the remainder of the paper, the

> 3 Health centers are the main first point of contact between the population and the health care system. These facilities provide both outpatient and inpatient care such as deliveries. At the tier above the health centers are the district hospitals followed by provincial hospitals and at the top tier are the country’s five referral hospitals. 4 To benchmark the gift values, it is estimated from household consumption data that in 2010 about 81 percent of the population lived on less than 3.10USD a day and 60 percent lived on less than 1.90USD a day ( <mark>iresearch.worldbank.org/PovcalNet).</mark>

> 5 Rwanda’s five provinces are divided into 30 districts, further divided into 416 sectors that are in turn divided into 2,150 cells encompassing the country’s 14,837 villages.

6 A sector typically contains a single public or non-for-profit faith-based health center with the catchment area of the health center corresponding to the boundary of the sector. However, there are some sectors that do not contain a health center and others have more than one.

4
treatment group is defined as the 100 sectors that implemented the demand-side incentives and the control group is defined as the 98 sectors that did not (see Figure 1).

A baseline survey was fielded from February to May 2010 to measure outcomes prior to the launch of the program and to establish internal validity of the experimental design. For a household level survey, twelve households with a woman aged 15-49 with a recent pregnancy or birth were selected from the catchment area of each of the 198 health centers. First, three cells (groups of villages) were randomly selected and four villages within each cell were randomly selected.<sup>7</sup> In each village, a field supervisor consulted the village leader and/or community health workers in order to identify the household with the most recent birth in each village. This resulted in a sample of 2,376 households.

A follow-up survey took place from November 2013 to June 2014.<sup>8</sup> The survey teams returned to the same villages sampled for the baseline and identified the households with the most recent birth or pregnancy following the same procedure as in the baseline survey.<sup>9</sup> Interviews were completed with 2,343 households. A rate of above 98.5% of successful interviews was achieved in all study arms. The follow-up survey also collected data on the 198 health centers whose catchment areas include the selected villages.

The primary outcomes for this analysis were constructed using data from interviews with the women who were pregnant shortly before the survey. A detailed questionnaire was administered to gather information on the care given to these women throughout their most recent pregnancy, delivery and post-partum. The analysis therefore relies on self-reported measures of service utilization.

The follow-up survey was fielded about nine months after the last funds for procurement of gifts were disbursed to the health centers. The survey was designed to not only evaluate the demand-side scheme but to collect data on the overall CPBF program. The termination of transfers to the health centers was unexpected and was not announced. As we will report below, almost all health centers in the treatment group reported they were still providing gifts as a policy although facing stock outs. Ideally, the data would have been collected while the program was still fully implemented.

To assess how the delayed data collection might affect the impact estimates and how the impact might have changed since implementation started, we complement the analysis conducted with the CPBF impact evaluation data with analysis of the Rwanda Demographic and Health Survey data collected from November 2014 to April 2015. The nationally representative survey collected detailed retrospective data on births in the five years preceding the survey. Timing of antenatal and postnatal care is reported by women for their most recent pregnancy. While the RDHS provides data on which district each survey cluster (enumeration area) is, we are uncertain about the exact sector. The GPS coordinates of clusters are randomly displaced to ensure confidentiality. Rural clusters are displaced up to five kilometers and urban clusters are displaced up to 2 kilometers.<sup>10</sup> Out of the 492 RDHS clusters, 304 are in the 19

> 7 During the fieldwork, the survey team learned that some villages, in five of the sectors, were served by health facilities different from the health center selected for the sample. In these cases, additional households in the villages covered by the selected cooperatives were added to the sample.

> 8 The duration of the follow-up survey was longer because in addition to a sample of women with recent pregnancies, the women interviewed in the baseline were tracked even if they relocated outside their baseline village.

> 9 In 32 villages, the same woman was interviewed both at baseline and follow-up surveys.

> 10 One percent of rural clusters displaced up to 10 kilometers

5
districts covered by the CPBF impact evaluation and 221 are recorded to be misplaced in one of the impact evaluation sectors. Of these 221 clusters, 109 are misplaced in treatment sectors and 112 in control sectors.

# Descriptive Statistics

## **i. Sample Characteristics and Balance**

Table 2 presents summary statistics of the baseline sample of women with recent pregnancies by treatment. It reports means of women’s characteristics and care provided during the most recent pregnancy. It also reports the p-value of t-tests assessing the similarity between treatment and control groups. The results confirm that the randomization achieved balance of observed characteristics between the two groups.

About 19 percent of the sample is from the South Province, 25 percent from the North, 28 percent from the West and 28 percent in the East. Households consisted of an average of five members and were located at an average distance of about 4.3 kilometers from the health center serving their village. The average age of the sample of women with recent pregnancy was 28.2 and 91 percent of them were married. Seventeen percent of the women have never attended school, 69 percent attended at least some primary school and the rest have attended at least some secondary or higher level education. Ninety percent were covered by Mutuelle, the community-based health insurance program.

The average number of births given by women in the sample is three and the women had on average 2.7 living children. Forty percent of the women reported ever using any modern method of family planning. Almost all women have received at least one antenatal consultation during their most recent pregnancy but only 63% initiated antenatal care within the first four months of the pregnancy and 37 percent received four or more consultations. Seventy-nine percent have delivered in a health facility, attended by a skilled health provider. Thirty-eight percent have reported receiving postnatal care within the ten days following the birth.

Some health centers endowed women with gifts even prior to the implementation of the intervention evaluated in this analysis. These health centers chose this type of conditional transfer as a strategy to increase coverage of services for which they were rewarded through the PBF scheme at the health facility level. About 5 percent reported receiving gifts for attending antenatal consultations, 4 percent for in-facility delivery and less than 2 percent for postnatal care. The values of these transfers are estimated to be less than a third of the values suggested for the CPBF in-kind transfers for each of the services.

The sampling strategy might have resulted in an unrepresentative sample of women with recent pregnancies. Because the selection of women was done with the aid of village leaders and community health workers, it could be that women in the sample are more likely to use health services relative to the overall population of pregnant women. A comparison of the sample to that of the nationally representative sample of the RDHS 2010, however, reveals that the baseline rates of utilization of maternal health services are similar in both surveys. The comparison was done with the subsample of 6
women who gave birth in the two years preceding the survey in the four provinces covered by our study (excluding the province of the capital city Kigali). The rates of women who received at least one antenatal consultation, initiated antenatal care within the first four months of their pregnancy and delivered in a health facility attended by a skilled health provider are identical at 98, 63 and 79 percent respectively. The rate of women who completed at least the suggested number of four antenatal consultations is 37 percent in our sample relative to 36 percent in the RDHS. The rate of women who reported receiving postnatal care within ten days after delivery, however, is much higher in our sample at 38 percent relative to 18 percent.

# **ii. Time Trends**

Between the time of the baseline and the follow-up surveys there has been an overall increase in coverage of the targeted health services independently of the demand-side conditional transfers. These trends can be best portrayed by comparing service utilization by the control group at baseline and follow-up. As presented in figure 2, the percentage of women who received their first antenatal consultation during the first four months of their pregnancy increased from 61.2 percent to 77 percent and that of women who received four or more consultations has risen from 37.2 to 43.4 percent. The rate of women who gave birth in health facilities increased from 78.4 to 94.6 percent.

Unlike the other rates, that of women who reported receiving postnatal care has decreased between the baseline and follow-up surveys. It is 39 percent in the baseline and 17 percent in the follow up. Given the discrepancy between our baseline survey and the RDHS 2010, we cannot conclude that there was an actual decrease in timely postnatal care initiation. According to the RDHS report there has been an increase in the rate of women who received postnatal care within the first six days after delivery from 18.5 percent in 2010 to 43.8 percent in 2015. It is important to note, though, that the way women were asked about postnatal care was different in the two surveys. The RDHS specifically asks about care provided immediately after delivery and those who report receiving such care are not asked about the timing of follow-up checks. The CBPF impact evaluation survey asks more generally about care after births as the program’s goal was to have women come back to the health centers after being discharged.

# **iii. Program Implementation**

Information on the program implementation was collected through the health centers and household surveys. As mentioned before, the last funding for the in-kind transfers was provided to health centers in February 2013. However, it was not announced that the program will terminate and some health centers still had available gifts (or resources for gifts) with which to endow the women who met the eligibility criteria. At the time of the follow-up survey, about 90 percent of the health centers’ in-charges in the treatment group reported that their facility usually provides women gifts if they meet each of the three targeted eligibility criteria. Among the control group, the percentages of health centers that reported providing gifts for timely ANC, delivery and timely PNC is 12, 16 and 9 percent respectively. It is important to remember that the health centers have the autonomy to independently implement such programs.

Although almost all health centers in the treatment group reported usually giving the gifts, most of them also reported experiencing stock outs of the gifts. Only 19 percent of the health centers reported not experiencing stock outs in the six months preceding the survey. Forty-nine percent reported 7
experiencing stock outs sometimes while 18 and 15 percent reported experiencing them frequently and very frequently respectively. When asked about the reason for the stock outs, 69 percent of the facilities mentioned lack of PBF funds as the main reason. Thirty percent stated issues with procurement or the suppliers as the main reason.

Out of the 956 women in the treatment arm who report receiving antenatal care in the first four months of their pregnancy, only 19 percent reported also receiving gifts. Twelve percent reported receiving a gift after delivering in a health facility and 7 percent reported receiving the in-kind transfers after receiving postnatal care within ten days after delivery. For each of these criteria, about 3 percent of the eligible women did not receive the gifts but were promised to receive them at a later stage. The rates of women meeting the eligibility criteria in the control group and receiving gifts from the health facilities where they received care are, 11, 4, and 1 percent for timely antenatal care, in-facility delivery and timely postnatal care, respectively. In all cases, the differences between the groups are statistically significant at the 1 percent level.

The follow-up survey also collected data on the women who were interviewed in the baseline survey. While these data are not used in the analysis presented in this paper, they enable us to calculate the rate of gift receipt for the different program implementation years, as more than half of the baseline sample had additional births since the baseline survey. The rates are highest for 2011 and gradually decrease in 2012 and 2013. For timely antenatal consultation, the rate of eligible women who received gifts is above 30 percent. It is important to keep in mind, however, that the estimate for 2011 is noisy, as few women had births two years in a row. In addition, the reporting of gifts received two years prior to the survey might be lower because of recall issues.

# Results

## **i. Analysis of the CPBF Impact Evaluation Data**

Given the random allocation of the intervention and the balance between treatment and control groups, the effect of the in-kind transfers can be estimated by simply regressing outcomes from the follow-up survey on an indicator for being from a treatment sector. Taking into account the sampling strategy, standard errors are clustered at the sector level. Table 3 presents regression results of this simple specification. The demand-side intervention has increased the rate of women receiving first antenatal consultation within the first four months of their pregnancy by about 7.7 percentage points from 73.1 percent among the control group to 80.8 percent among the treatment group. The effect is statistically significant at the 1% level. The rate of women who received postnatal care within ten days after delivery has increased by 8.6 percentage points from 12.2 percent among the control group to 20.8 percent among the treatment group. Also this effect is significant at the 1% level. No statistically significant impact is detected on in-facility deliveries. The rate of such deliveries is 95 percent in the control group and 94 percent in the treatment group.

Timely antenatal and postnatal care, as well as in-facility deliveries, were directly incentivized by the program. Table 3 shows also results of the impact of the intervention on related but non-targeted outcomes. Although it increased the rate of women who commenced their antenatal care early, the 8
analysis does not show a statistically significant increase in the rate of women who received any antenatal care or that of women who received four or more consultations. It is important to stress, though, that the rate of women receiving any antenatal care is practically universal. The impact parameter for four or more ANC visits is positive with a p-value of 0.146. With respect to receiving any postnatal care, however, the program is associated with an increase of nine percentage points. The effect is significant at the 1% level.

Table A1 in the appendix presents results of a different model specification in which controls are added to the regressions. Those are women’s characteristics (age, marital status, education, and birth order), household characteristics (education of the head of the household and number of household members), an indicator for the household being more than four kilometers away from the health center and district dummies. About 10 percent of the sample is lost for having missing information for at least one of the controls. However, the estimated treatment coefficients are almost identical in both magnitude and their statistical significance.

The results presented in Table 3 are from an analysis conducted without using sample weights. While they represent an unbiased estimate of the impacts given the experimental design, the impacts might not be representative given the sampling strategy. Table A2 in the appendix presents results for the same regression specifications as in Table 3 but with weighting observations according to the number of households per sector. The weighting causes negligible changes in the estimated parameters.

# **ii. Analysis of the Rwanda Demographic and Health Survey 2014-15 Data**

Unlike the CPBF impact evaluation survey that interviewed women with the most recent birth in each visited village, the RDHS 2014-15 collected detailed information on pregnancies in the five years preceding the survey. Since the RDHS cluster locations are reported with error, we are uncertain about the sector to which each cluster belongs and therefore about exposure to treatment. In our regressions, we use two classifications of exposure to the demand-side in-kind transfer program. In the first, we regress the outcome indicators on a binary variable for the location of the cluster being misplaced in the area of a treatment sector. For a second approach, we calculated the probabilities that each cluster is in a treatment sector. Using the displacement distances as radiuses, we draw a circle around each misplaced cluster and calculate what percentage of the circle covers a treated sector. We then regress the outcome indicators on treatment probability. The sample sizes for these regressions are larger as we can include all 304 cluster in the 19 districts and not only the 221 clusters misplaced within the study sectors. In both approaches we interact treatment (either the binary variable or treatment probabilities) with year of birth. We control for the age of the woman at the time of birth, her marital status, education level, birth rank, an indicator for a rural DHS cluster and district dummies. The RDHS sample weights are used and standard errors are clustered according to the DHS cluster.<sup>11</sup> The results of this analysis should be taken as suggestive only. First, the treatment coefficients represent underestimates of the actual impacts. Under the assumption that the intervention has a non-negative

> 11 Clustering standard errors at the sector level results in negligible difference in results, as the majority of clusters do not have other clusters displaced in the same sector.

9
impact, having some treatment clusters classified as control clusters (or with treatment probability smaller than one) will increase coverage rates in the control group. Similarly, coverage rates in the treatment group will be lowered by having control clusters classified as treatment ones. As a result, the estimated difference between the groups will be smaller than the actual difference. Second, detailed information was only provided on the last pregnancy of each respondent. Therefore, the data are not representative by year of birth.

Figure 3 presents the annual impact coefficients of the in-kind transfers on timely antenatal care. We restrict the sample to births after June 2011 to assure women were exposed to the intervention for the entire duration of their pregnancy. The left frame shows the results of the regression using the binary treatment indicator and the right frame the results when using treatment probabilities. The annual treatment coefficients are plotted with the 95 percent confidence intervals. In both specifications, the treatment coefficients reduce with the duration of implementation and the 2011 treatment coefficient is statistically significant. The treatment coefficients for 2012 are significant at the 10 percent level in the specification using the binary treatment indicator and at the 5 percent level in the other specification. For all other years, we cannot reject the coefficients equaling zero and they are estimated to be negative for 2014 and 2015. These results suggest that the estimated impact found in our primary analysis might have been larger if the follow-up survey was fielded earlier when health centers were more likely to have stocks of gifts to reward eligible women.

As in the analysis using the CPBF impact evaluation data, we do not find any impact on in-facility deliveries and we do not present the results for brevity. As for postnatal care, we are unable to construct a measure comparable to the one we are using in our primary analysis with the RDHS data. The goal of the program was to bring women back to the health centers after they were discharged following their deliveries. For women who received postpartum care immediately after delivery, the RDHS now collects data on the timing of additional postnatal consultations. More than 50 percent of the women recorded to receive postnatal care are reported to receive it within one hour after delivery. Using the timing of care according to the RDHS will imply rate of 45 percent of postnatal within ten days after delivery in comparison to 17 percent in the impact evaluation data. For comparison, rates of timely antenatal care are 77 percent in both data sets.

# Discussion

We find that conditional demand-side in-kind transfers can be effective in increasing uptake of health services. The scheme that was introduced as part of the Rwanda Community Performance-Based Financing program caused a fairly substantial increase in the rate of women who received timely antenatal and postnatal care. We are able to assign a causal impact of the demand-side transfers even though there were other interventions or factors that increased maternal health service utilization as portrayed by the increase in health service utilization by the control group.

The demand-side incentives improved indicators that were not impacted by incentivizing health providers through the Performance-Based Financing scheme at the health center. However, we cannot compare the effectiveness of incentivizing health providers versus only incentivizing users of the health care system as the demand-side transfers were introduced on top of the countrywide facility-level PBF 10
scheme. It could be that there were synergies between the incentives to the facilities and the users. It is noteworthy though that we find a difference in outcomes between the treatment and control groups although some health centers could independently endow women with gifts in order to increase their performance with respect to incentivized indicators. As the number of countries in Sub-Saharan Africa that are introducing or scaling up pay-for-performance schemes to incentivize health providers is rapidly increasing (World Bank 2015), this study shows that incentivizing users can enhance the impact of these schemes on service utilization.

Although the results are consistent with previous studies of schemes that incentivize health service utilization with conditional transfers, they are remarkable considering that most eligible women reported not receiving the transfers.<sup>12</sup> It is possible that women went to the health centers with the expectation to receive the gifts and only learned about the lack of gifts while already receiving the services. Although funding provided to health centers to procure the gifts stopped, the program itself was not announced to be terminated, so both health providers and users could have been under the impression that the program was ongoing. Our analysis of the RDHS 2014-15 data suggests that the estimated impact might have been larger if the health centers were not experiencing stock outs.

> 12 This is partially due to the timing of the follow-up survey months after the last funding was transferred to the health centers.

11