metadata: authors: 0: author: Obare,Francis
1: author: Okwero,Peter
2: author: Villegas,Leslie
3: author: Mills,Samuel Lantei
4: author: Bellows,Ben
count: Uganda
docna: 0: docna: Increased coverage of maternal health
            services among the poor in western Uganda in an output-based
            aid voucher scheme
docty: Policy Research Working Paper
subtopic: Taxation & Subsidies,Health Care Services Industry,Health Service Management and Delivery,Inequality
teratopic: Macroeconomics and Economic Growth,Industry,Health, Nutrition and Population,Poverty Reduction
abstracts: cdata!: Vouchers stimulate demand for health
            care services by giving beneficiaries purchasing power. In
            turn, health facilities’ claims are reimbursed for providing
            beneficiaries with improved quality of health care.
            Efficient strategies to generate demand from new, often
            poor, users and supply in the form of increased access and
            expanded scope of services would help move Uganda away from
            inequity and toward universal health care. A reproductive
            health voucher program was implemented in 20 western and
            southwest Ugandan districts from April 2008 to March 2012.
            Using three years of data, this impact evaluation study
            employed a quasi-experimental design to examine differences
            in utilization of health services among women in voucher and
            nonvoucher villages. Two key findings were a
            16-percentage-point net increase in private facility
            deliveries and a decrease in home deliveries among women who
            had used the voucher, indicating the project likely made
            contributions to increase private facility births in
            villages with voucher clients. No statistically significant
            difference was seen between respondents from voucher and
            nonvoucher villages in the use of postnatal care services,
            or in attending four or more antenatal care visits. A net
            33-percentage-point decrease in out-of-pocket expenditure at
            private facilities in villages with voucher clients was
            found, and a higher percentage of voucher users came from
            households in the two poorest quintiles. The greater uptake
            of facility births by respondents in voucher villages
            compared with controls indicates that the approach has the
            potential to accelerate service uptake. A scaled program
            could help to move the country toward universal coverage of
            maternal health care.
geo_reg_key: 517191
repnme: repnme: Increased coverage of maternal health
            services among the poor in western Uganda in an output-based
            aid voucher scheme
display_title: Increased coverage of maternal
            health services among the poor in western Uganda in an
            output-based aid voucher scheme
listing_relative_url: /research/2016/06/26506638/increased-coverage-maternal-health-services-among-poor-western-uganda-output-based-aid-voucher-scheme
topicv3: Poverty m1326687 695,Sexual Reproductive, Maternal and Child
            Health m1327262 639,Health Services Delivery m1327294 1305,Health m1504661 3480,Vouchers m1504820 3553
publishtoextweb_dt: 2016-06-20T00:00:00Z
historic_topic: Macroeconomics and Economic Growth,Industry,Health, Nutrition and Population,Poverty Reduction
no_of_pages: 46
author_keys: 0: author_key: 428025
1: author_key: 450582
2: author_key: 743578
model_extractions: WPS7709 Policy Research Working Paper 7709 Increased Coverage of Maternal Health Services among the Poor in Western Uganda in an Output-Based Aid Voucher Scheme _Francis Obare Peter Okwero Leslie Villegas Samuel Mills Ben Bellows_ Health Nutrition and Population Global Practice Group & Social, Urban, Rural and Resilience Global Practice Group June 2016
Policy Research Working Paper 7709

# **Abstract**

Vouchers stimulate demand for health care services by giving beneficiaries purchasing power. In turn, health facilities’ claims are reimbursed for providing beneficiaries with improved quality of health care. Efficient strategies to generate demand from new, often poor, users and supply in the form of increased access and expanded scope of services would help move Uganda away from inequity and toward universal health care. A reproductive health voucher program was implemented in 20 western and southwest Ugandan districts from April 2008 to March 2012. Using three years of data, this impact evaluation study employed a quasi-experimental design to examine differences in utilization of health services among women in voucher and nonvoucher villages. Two key findings were a 16-percentage-point net increase in private facility deliveries and a decrease in home deliveries among women who had used the voucher, indicating the project likely made contributions to increase private facility births in villages with voucher clients. No statistically significant difference was seen between respondents from voucher and nonvoucher villages in the use of postnatal care services, or in attending four or more antenatal care visits. A net 33-percentage-point decrease in out-of-pocket expenditure at private facilities in villages with voucher clients was found, and a higher percentage of voucher users came from households in the two poorest quintiles. The greater uptake of facility births by respondents in voucher villages compared with controls indicates that the approach has the potential to accelerate service uptake. A scaled program could help to move the country toward universal coverage of maternal health care.

This paper is a product of the Health Nutrition and Population Global Practice Group and the Social, Urban, Rural and Resilience Global Practice Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at pokwero@worldbank.org, and lvillegas@ worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
**Increased Coverage of Maternal Health Services among the Poor in Western Uganda in an Output-Based Aid Voucher Scheme**

Francis Obare, Population Council Peter Okwero,* The World Bank Leslie Villegas,* The World Bank Samuel Mills, The World Bank Ben Bellows, Population Council _Key words_ : impact evaluation; vouchers; reproductive health; output-based aid; Uganda _JEL classification_ : I14 _Sector Board_ : Health Global Practice

# **----------------------------**

*Corresponding authors: Peter Okwero, The World Bank, Kampala, Uganda. Tel. +256 414 302227, Mobile: +256 772 636486, e-mail: pokwero@worldbank.org; and Leslie Villegas, The World Bank/GPOBA, Washington, DC, USA. Tel. +12024735265, e-mail: lvillegas@worldbank.org.
# **<<A>>** Acknowledgments

We thank Venture Strategies for Health and Development (VSHD), Marie Stopes International Uganda (MSI), Mbarara University of Science and Technology, Population Council, and the University of California at Berkeley for conducting the baseline and endline household surveys and secondary studies reported in this working paper. The GPOBA and Bill and Melinda Gates Foundation are gratefully acknowledged for their financial support of research activities and the data analysis of baseline and endline surveys. The authors are grateful to the peer reviewers: David Evans, Damien de Walque, Edit V. Velenyi, Ashis Das, Huihui Wang, Gandham N.V Ramana, and David Griffith for their valuable comments. Olusoji Adeyi and Magnus Lindelow (Health Global Practice) provided overall guidance.

2
# **<<A>>** Acronyms and Abbreviations

|AIDS|Acquired immunodeficiency syndrome|
|---|---|
|ANC|Antenatal care|
|BulkSMS|Web-based bulk text-messaging platform|
|CAS|Country assistance strategy|
|CPIs|Client-provider interactions|
|DHS|Demographic and Health Survey|
|GPOBA|Global Partnership on Output-Based Aid|
|GPS|Global positioning system|
|HFA|Health facility assessment|
|HNP|Health, nutrition and population|
|IMF|International Monetary Fund|
|IVEA|Independent verification and evaluation agent|
|KfW|Kreditanstalt für Wiederaufbau, German Development Bank|
|MDG|Millennium development goal|
|MSI|Marie Stopes International|
|NGO|Nongovernmental organization|
|OBA|Output-based aid|
|PNC|Postnatal care|
|PPH|Postpartum hemorrhage|
|RHAP|Reproductive health action plan|
|RHVP|Reproductive health voucher program|
|SD|Safe delivery|
|SP|Service provider|
|STD|Sexually transmitted diseases|
|STI|Sexually transmitted infection|
|UBOS|Uganda Bureau of Statistics|
|UHC|Universal health coverage|
|UNDP|United Nations Development Program|
|USAID|United States Agency for International Development|
|VCBD|voucher community-based distributor|

3
|VMA|voucher management agency|
|---|---|
|VSHD|Venture Strategies for Health and Development|
|WTP|Willingness to pay|

Currencies are in US dollars ($) unless stated otherwise.

4
# **<<A>>** Introduction

The terms used to describe the combined use of vouchers and output-based contracting include _output-based aid_ (OBA), _demand-side finance_ , and _voucher and accreditation_ programs (Gorter, Sandiford, Rojas, & Salvetto, 2003; Janisch & Potts, 2005; Musgrove, 2011). OBA is a form of results-based financing, which links payments to verified delivery of specific health outputs or outcomes. Health-sector staff in traditional salaried positions may have little incentive to raise their productivity or be concerned with client perceptions of health care quality. OBA subsidies, however, create incentives to improve the efficiency of health services delivery and increase access to important health services for new users. Vouchers stimulate demand for health care services and give the poor the purchasing power to seek care from the full range of available service providers (SPs). Voucher programs have the potential to improve health care and health outcomes at the facility level and among the general population.

Several countries have successfully employed OBA subsidies as a means to create incentives to improve the efficiency of health care provision, deliver health services to low-income populations, and increase access to important health services for new users (B. W. Bellows et al., 2013; N. Bellows, 2011). The Taiwan, China, Voucher Program was the first large-scale outputbased voucher subsidy for health care in a low-income country in the 1960s and 1970s. The program offered male and female sterilization services at government and private facilities for low-income couples and couples with two or more children (Cernada and Chow 1969). The Republic of Korea implemented a similar contemporary program (Ross et al. 1970). Twenty years later, in the mid-1990s, Nicaragua implemented two voucher programs to prevent the spread of STIs among commercial sex workers and adolescents (Borghi, Gorter, Sandiford, & Segura, 2005; Meuwissen, Gorter, Kester, & Knottnerus, 2006a, 2006b; Meuwissen, Gorter, & Knottnerus, 2006). Ten years later, in the mid-2000s, the number of programs in Africa and South Asia grew, including the Gujarat SD voucher program, which subsidized access to private SPs for pregnant women living below the official poverty line (Bhat, Mavalankar, Singh, & Singh, 2009; Mavalankar et al., 2009). It should be noted that evidence in a recent study suggests the Gujarat program did not produce noticeable population effects (Mohanan et al., 2014). In 2006 two RHVPs were launched in East Africa, representing a significant new approach to providing pro-poor 5
health-care subsidies in the region (Abuya et al., 2012; B. Bellows, Kyobutungi, Mutua, Warren, & Ezeh, 2013; Obare, Warren, Abuya, Askew, & Bellows, 2014).

Recently growing awareness of health inequalities has sparked increased calls for a progressive, pro-poor expansion of national health systems to move toward UHC (Gwatkin & Ergo, 2011). Although the shape of UHC will vary between countries, the movement is defined by common objectives of incremental improvement in quality, reducing out-of-pocket expenditure at the point of care, and extending access to populations that lack it. Calls for progressive universalism argue that coverage should be extended first to those who are least likely to use services in the absence of the extension. Voucher programs are designed to offer a narrow health care package to a well-defined, disadvantaged population lacking access. In such populations, improved uptake should be observed in the local population within a relatively short time frame, depending on the nature of the service and the degree to which financial barriers constrained earlier access to care.

The goals of voucher programs are to reduce the financial barriers to accessing services for poor and underserved populations, reduce inequality in service use, improve quality of care, achieve cost-effectiveness in service delivery, and improve health outcomes (Bhatia and Gorter 2007; Cave 2001; Gorter et al. 2003). The programs aim to achieve these goals through various mechanisms. The programs, for instance, subsidize services and put in place mechanisms for identifying beneficiaries to ensure that target populations are reached. The programs also employ explicit, performance-based contracting, which, in theory, requires SPs to meet set minimum standards of care before being accredited. It is further expected that accreditation of several SPs should stimulate competition for voucher clients with pressure to improve service quality. The programs also negotiate reimbursements to SPs to maintain costs, which, together with the set minimum standards of care, should ensure cost-effectiveness in service delivery.

This paper presents the results of a quasi-experimental evaluation of the Uganda RHVP and findings from other studies including health worker job satisfaction and trends in out-of-pocket spending on maternal health services. The paper presents outcomes in four broad categories: knowledge, behavior (including utilization and access), quality, and out-of-pocket spending as commonly presented in the literature (Bellows, Bellows, and Warren 2011; Brody, Irige, and Bellows 2015). A fifth category, health status, is not addressed in this study as the sample size required to detect a noticeable difference in mortality or morbidity was beyond the evaluation scope. Future studies could consider undertaking such an endpoint or present results of modeling 6
under a range of assumptions drawing from the literature linking service uptake with health outcomes (Tura, Fantahun, and Worku 2013).

In studies of vouchers, knowledge outcomes can be measured among facility clients, SPs, and the general population. Common metrics include knowledge of disease symptoms, program characteristics (i.e., where to find a voucher or clinic location), and SP adherence to patient safety and treatment guidelines. Improvements in behaviors in the context of a health systems intervention largely encompass health-seeking behaviors. Other barriers to health care—such as distance to health facilities, poor roads, and difficulty in securing transport—are taken into account because they may keep clients from seeking care. However, if cost is the principal barrier, it follows that use of the facilities should increase with voucher distribution. If the burden of the untreated health conditions is high among the general population, it may be possible to detect a change in utilization patterns among the general population following the introduction of vouchers. Quality is measured by improvements in facility infrastructure, in the service delivery process (i.e., how clients are medically treated), and in client satisfaction with services that also contribute to reductions in maternal and neonatal morbidity and mortality.

Cost metrics are another important area to monitor. To gain insight into whether OBA subsidies are effective in improving health care delivery and health outcomes, it is important to monitor client out-of-pocket spending, facility revenue and costs, the ratio of program subsidies to the number of clients served, and related expenses. The final area to monitor is population health. The incidence of new cases per month or quarter in a population served by contracted facilities may be one measure of health outcomes. The odds of exposure in clinic-based cases and controls may be another approach. The change in prevalence in a difference-in-differences design may constitute yet another metric that makes it possible for administrators, funders, and other stakeholders to know whether OBA subsidies are a success.

# **<<A>>** Country Context

Uganda is among the world’s poorest countries, with a GDP US$6777 and a life expectancy of 59 years (World Development Indicators 2014). According to the 2013 United Nations Development Program (UNDP) Human Development Report, Uganda is ranked 161 out of 185 countries (UNDP 2013). Although the country is on track to achieve the Millennium Development Goal (MDG) targets of halving poverty, improving gender equality, and empowerment of women, it is significantly lagging on reducing child mortality (MDG 4) and improving maternal health (MDG 7
5) and, without accelerated progress, is unlikely to achieve the MDG targets by 2015. Although the proportion of deliveries at facility has improved, maternal mortality remains a significant challenge, particularly for disadvantaged populations. The national maternal mortality ratio in the 2011 Uganda Demographic and Health Survey (DHS) was statistically unchanged at 438 deaths per 100,000 live births compared to the 2006 DHS. Perinatal and maternal morbidity and mortality remain major causes of the high national disease burden, accounting for 20.4 percent of the burden.<sup>1</sup> Hemorrhage, abortion, sepsis, obstructed labor, pregnancy-induced hypertension, and malaria make up the major causes of maternal deaths and stillbirths. Owing to poor-quality services and limited access to skilled delivery services, compounded by the high cost of care, in the five years prior to the 2011 DHS, 43 percent of pregnant women delivered at home without the assistance of skilled health personnel with many of the home births disproportionately occurring among the poor and in rural areas. The provision of comprehensive emergency obstetric care in particular faced many challenges.

The need to reverse poor maternal health outcomes was and remains a key priority for Uganda. The four main strategies to reverse poor maternal health outcomes highlighted in the Road Map for Acceleration of Maternal and Neonatal Mortality and Morbidity (2006–15) include (1) improving and expanding the quality of maternal and newborn care, (2) improving access to family planning services, (3) providing adolescent reproductive health services, and (4) strengthening supply-chain management for reproductive health commodities in the public and private sectors. In this respect, the government, in collaboration with the World Bank and Kreditanstalt für Wiederaufbau (KfW, German Development Bank), designed the RHVP, building on the earlier STI OBA voucher scheme also financed by KfW.

The RHVP was in line with the World Bank’s Country Assistance Strategy (CAS) and Reproductive Health Action Plan (RHAP). Uganda is a priority country in the World Bank’s RHAP, which aims to address high fertility, reduce the unmet need for contraception, improve pregnancy outcomes, and reduce STIs.<sup>2</sup> The 2011–15 Uganda CAS mentions the RHVP as one of the expected results under CAS Outcome 3.2 (“Strengthened Health Care Delivery”). The project also contributed to the goal of the national Safe Motherhood Program of “ensuring that no woman or newborn dies or incurs injuries due to pregnancy and/or childbirth.” The project focused on reimbursing results and was consistent with the World Bank’s Health Nutrition and Population Strategy. The project addressed not only MDG 5—reducing maternal mortality—but also MDG 4—improving child health by reducing perinatal deaths—both being health problems of national 8
importance in Uganda. The subsidies were expected to reduce the financial barriers to care and support the provision of services that reduce poor women’s pregnancy and childbirth risks.

# **<<A>>** RHVP Implementation

The RHVP in Uganda was initiated in 2006 when the STI voucher service was launched. The program started as a pilot in four southwestern districts (Mbarara, Kiruhura, Ibanda, and Isingiro). Known as HealthyLife, it was funded by KfW, and implemented between 2006 and 2008 to subsidize diagnosis and treatment for STIs. Clients presenting with STI complaints could purchase the vouchers at a cost of USh3,000 (approximately $1.50) from selected local general retailers, drug shops, and pharmacies. The vouchers were then redeemed for services at private for- and notfor-profit facilities contracted by the program. They entitled the client and one sexual partner to a total of four visits each to ensure effective treatment. Facilities were reimbursed, on average, at $10 per STI client.

The findings of the pilot in the four districts fed into the expansion of the RHVP for maternal health voucher service. The program expanded with additional funding from the Global Partnership on Output-Based Aid (GPOBA–World Bank) and KfW to cover STI treatment and safe delivery (SD) services in 20 districts in the western and southwestern regions from April 2008 to March 2012. The maternal health voucher, known as HealthyBaby, cost USh3,000 (approximately $1.50) for subsidized safe motherhood services to economically disadvantaged women. The benefit package included four ANC visits, health facility delivery (normal, complicated, or Caesarean), transport in cases of emergency to a referral facility, treatment and management of complications, PNC for up to six weeks. Facilities were reimbursed $25 for normal deliveries and $79 for complicated deliveries including C-section. Community-based distributors, who were responsible for administering a district-customized poverty grading tool, educated poor pregnant women on the importance of seeking prompt ANC visits and a facility delivery. For women who qualified based on the poverty grading tool, the distributor sold the voucher for free maternal health services. The STI voucher continued to be distributed to all clients with STI complaints at drug shops and pharmacies. For the safe motherhood voucher, the poverty tool consisted of eight items of household assets and amenities, expenditure or income, and access to health services. Women scoring between zero (the minimum) and 15 points qualified for the voucher. The proportion of women reached by the voucher program varied from community to community. On average, the program covered 38 percent of all births to the bottom 40 percent of 9
the population in the targeted districts (Kanya et al. 2013). Accredited SPs were selected from a pool of private for- and not-for-profit SPs offering basic or comprehensive emergency obstetric care.

The three main implementing roles were the fiduciary agent, voucher management agency (VMA), and independent verification and evaluation agent (IVEA). KfW served as the fiduciary agent and had management and fiduciary responsibility (procurement and financial management) over the project. Contracted by the fiduciary agent, MSI served as the VMA. Its main functions were the following: (1) contract and supervise voucher SPs to provide SD and STI services, (2) design vouchers, (3) develop claims-processing software, (4) develop a comprehensive behavioral change campaign and voucher marketing strategy, (5) enlist voucher community-based distributors (VCBDs) to sell vouchers to eligible poor women,<sup>3</sup> (6) train SPs, vendors, and project staff, and (7) perform claims processing and contract management. PricewaterhouseCoopers Limited served as the IVEA and carried out the following functions: (1) verification of project outputs at the facility level, and (2) assessment of the effectiveness of the project systems and processes, including outreach and clinical activities.

Policy champions from the Uganda National STD/AIDS Control Program and the commissioner for reproductive health were instrumental in the initial establishment of the voucher program. They not only popularized the program within government circles but also worked closely with the VMA to ensure that its design was in line with the Ministry of Health’s policy objectives. A private consulting firm was contracted to map SPs before joint selection and accreditation by the VMA and the consulting firm. Facilities were accredited based on minimum standards of care (Class A) and accessibility of services. Facilities that failed to meet minimum standards but were located in underserved areas were considered for Class B contracting, in which it was expected that SPs would use the income generated through the program to upgrade the facility and improve quality of care over time. An effort was made to link such facilities to referral centers for emergency care of complications (PS Consulting 2009, 1–14).

The VMA used multiple marketing campaigns to reach a large number of potential clients to increase awareness of STI symptoms, the importance of delivering at a health facility, and the voucher program itself. The strategy included extensive radio campaigns, market day visits, community film nights, and sponsored events such as local concerts. The HealthyLife and HealthyBaby programs were branded, using colorful logos. The program further involved testing the use of mobile phone technology to communicate with the contracted health care facilities (SPs) 10
through BulkSMS, a web-based bulk text-messaging platform. The VMA used BulkSMS’s service for program administration, including notifying and confirming payments with SPs, making program announcements, coordinating site visits, and confirming acceptance of contractual changes (Densmore 2012a).

SPs were reimbursed for services rendered through electronic transfer of funds to their bank accounts. This was done after the VMA had verified the submitted claims for compliance with program regulations and guidelines. The verification process went through various stages. Initially, the VMA outsourced claims processing to a private insurer, Microcare. The database that was developed by the company was intended to flag questionable claims for manual review. However, the system identified a very high proportion of problematic claims, which reduced the usefulness of the program because a great majority of claims required manual review. By late 2006 the regional VMA office in Mbarara had hired a medical expert to vet the claims. When the program was expanded in 2009, a new system—claims processing database management system— was developed (Densmore 2012b, 1833–42). In addition, vetting teams carried out spot checks on samples of claims for compliance with medical and financial standards in an effort to control fraud. In cases where fraud was evident or highly probable, the claims were rejected and the SPs were paid a fraction of the claimed amount or were suspended from the program.

# **<<A>>** Evaluation Objectives

The RHVP program design included an impact evaluation. The evaluation of the SD component is reported in this paper. The evaluation had two main objectives: (1) assess the effect of the program on improving access to, and the quality of, reproductive health services while reducing inequities in their use; and (2) evaluate the impact of the program on improving reproductive health behaviors and outcomes at the population level. The evaluation aimed to assess the effect of the program on targeting beneficiaries; improving knowledge of reproductive health services; facilitating health service use and access, particularly among new users; assessing equity; ascertaining costs (service costs, program costs, and out-of-pocket expenses); and ensuring quality of care (clinical quality, client and SP satisfaction). In 2008 GPOBA contracted Venture Strategies for Health and Development (VSHD) to undertake a population-based evaluation of the program.

# **<<A>>** Evaluation Design

11
The evaluation of the voucher program’s efficacy adopted a retrospective quasi-experimental design using the second of two rounds of data collection, which included respondents drawn from a group exposed to the voucher and a comparison group. The design was informed by the fact that the intervention sites were not randomly assigned. The original 2008 design incorporated a prospective cluster randomized study that called for the systematic placement of voucher distributors—with a random start at one cluster—in half of the 22 clusters of local administrative units (parishes) to ensure an even distribution of voucher- and nonvoucher-exposed communities. Each of the 22 clusters centered on a voucher-contracted facility. The target population, drawn from Bushenyi, Ibanda, Isingiro, Kiruhura, Kamwenge, and Mbarara districts, consisted of women who were between 15 and 49 years of age and had had a pregnancy during the previous 12 months. Respondents were selected from 22 clusters within five to ten kilometers of contracted facilities and three kilometers from a main road. It was assumed that individuals located within those ranges had similar levels of access to the contracted facilities. A two-stage design was used to identify potential study participants. The first stage was a simple random sample of villages within each cluster, and the second stage was a census of households with recent pregnancies within selected villages. The sampling frame used maps and population figures from the Ugandan Bureau of Statistics (UBOS) 2002 census.

A sample of 2,627 women was to be selected from 84 villages in the 22 clusters. Using the national rate of 5.2 deliveries per 100 total populations per year, the average number of pregnant and recently delivered women in all villages within each cluster was estimated to be 28 women per village. Assuming a baseline proportion of 30 percent facility deliveries, it was estimated that 120 pregnant and recently delivered women would be needed in each cluster to detect a change of 2 percent in the proportion of facility-based births between the baseline and a one-year follow-up among the treatment groups for a two-sided T-test at 5 percent significance and 90 percent power (120 deliveries * 22 clusters = 2640 female respondents).

Based on the target of 120 deliveries in each cluster and using the UBOS census figures to estimate village populations, the necessary number of villages was selected from a list of villages in each cluster, limiting the selection to villages with 20 or more estimated deliveries a year. The additional 467 women included in the sample size were intended to account for nonresponses.

However, a deviation from the original plan occurred immediately after the baseline survey was carried out because of insufficient communication between the researchers and the program operations team. The program operations team had recruited voucher distributors in villages 12
designated as controls, which contaminated the original evaluation design. Nonetheless, a followup survey was carried out in late 2010 and early 2011 in the six districts (Bushenyi, Ibanda, Isingiro, Kamwenge, Kiruhura, and Mbarara) where the SD voucher service had first been launched.

Both baseline and follow-up surveys used a two-stage cluster sample design. First, geographic data obtained from the Uganda Bureau of Statistics (UBOS) were used to identify parishes within six to ten kilometers of 13 facilities that had been contracted to provide services to voucher clients and within three kilometers of a major road. Parishes were then randomly selected from among those within the stipulated distance to the facilities. In the second stage, villages were randomly selected from the sampled parishes. At baseline, 58 parishes were randomly selected for inclusion in the study. There were 231 villages within these parishes with populations ranging from 75 to 1,803 inhabitants, giving a total of 102,260 persons, according to the 2002 census. A total of 94 villages were randomly selected for inclusion in the baseline survey study.

In the follow-up survey, 75 parishes were randomly sampled, and 133 villages from these parishes were selected for inclusion in the study. Of the sampled villages, 68 had been included in the 2008 survey, and 65 were sampled from within five kilometers of the contracted facilities to maximize the possibility of contacting respondents likely to have used the vouchers. This approach was adopted because the 2010 voucher claims data showed a very high concentration of clients in parishes within five kilometers of contracted facilities. Of the 94 baseline and 133 endline villages, none of the baseline and 26 of the endline villages were within five kilometers, and all 94 of the baseline and 68 of the endline villages were within six to ten kilometers.

Besides the population surveys, cross-sectional health facility assessment (HFA) data were collected in 2010 through a number of techniques, including observations of client-provider interactions (CPIs) during consultations, client exit interviews, interviews with SPs, record reviews, facility inventories, and service statistics. The HFA was conducted in 20 contracted (ten offering STI and ten offering safe motherhood services) and ten comparison facilities. The facilities were specifically selected based on the type of services offered and their size in terms of client volumes. This report uses data from CPIs and interviews with SPs. Additional crosssectional data were obtained from (1) voucher sales and claims up to mid-2011, provided by the VMA, (2) semistructured interviews conducted in August 2010 with SPs from contracted and noncontracted facilities to examine workload and the level of job satisfaction among the SPs, and (3) cost data for the voucher program, obtained from the VMA in July 2011.

13
# **<<B>>** Household Surveys (Primary Data)

The target populations in both the baseline and follow-up household surveys were women aged between 15 and 49 who had experienced a pregnancy or birth during the 12 months prior to the survey as well as men of similar age group whose partner was pregnant or had given birth over the same period. Two visits were made to each village. The purpose of the first visit was to seek the cooperation of the local council chair in generating a list of households in which a pregnancy or birth had occurred in the previous 12 months and to take global positioning system (GPS) coordinates to ensure that all the villages were within the prescribed geographical location. During the second visit, a survey was administered to all women living in those households meeting the inclusion criteria. A total of 2,266 women and 177 men participated in the baseline survey, and another 2,313 women and 582 men participated in the follow-up survey.

In both surveys, respondents provided information on household assets and amenities, health-related household arrangements, food security, household expenditures on goods and services, individual background characteristics (age, education level, religious affiliation, and marital and employment status), general health status and health care utilization, childbearing experiences and intentions, family planning knowledge and use, trust and social cohesion in the community, and awareness, use, and perceptions regarding vouchers. Also sought were men’s perceptions regarding the importance and timing of ANC, delivery care, and PNC for their partners, and their willingness to pay (WTP) for such services. However, findings from the men’s WTP questions are not presented here. In 2008 women were asked detailed questions about their two most recent births, including the use of ANC, delivery care, and PNC, and experiences and management of any complications. In the follow-up survey, women provided detailed information on all births in the five years prior to the survey. Table A1 in the appendix presents the distribution of participants in the surveys (i.e., the 2008 baseline survey and 2010–11 follow-up survey) by various background characteristics.

Written informed consent was obtained from participants in both surveys, and the Institutional Review Boards of the Population Council and Mbarara University granted ethical clearance for the surveys.

14
# **<<B>>** HFA (Primary Data)

Observations of CPIs were done for clients seeking ANC for the first visits (under 24 weeks) and last visits (36 weeks or more); PNC within 48 hours, two weeks, and four to six weeks; STI treatment; and family planning services. Although family planning was not initially one of the services subsidized by the maternal health voucher program, a follow-up program to subsidize the services had already been initiated through additional funding from the United States Agency for International Development (USAID), hence the rationale for conducting CPIs for family planning. Trained nurses conducted the CPIs. They observed both the process (how clients were treated and whether they actively participated) and the content (what clients were told, technical competence of the SP, accuracy of information, and provision of essential information) during consultation. In both contracted and comparison facilities, six clients were selected for each ANC and PNC component, as well as family planning services in each facility. At facilities with more than six clients at the time the research assistants visited, six clients were selected at random. Written informed consent was obtained from the clients before conducting the CPIs. Table A2 in the appendix presents the distribution of CPIs that were conducted using various background characteristics for each of the services.

Interviews with SPs targeted those working in the maternal and child health and family planning units in the facilities included in the 2010 HFA. All SPs in these units were eligible for the interviews. Information was obtained about their knowledge, attitudes, and practices regarding reproductive health issues, including ANC, delivery care, PNC, STIs and HIV, family planning, gender-based violence, and vouchers. Either the nurses or the social scientists conducted the interviews after obtaining written informed consent from the SPs. Table A3 presents the distribution of SPs who were interviewed based on various background characteristics. Similar to the household surveys, the Institutional Review Boards of the Population Council and Mbarara University granted ethical clearance for the HFA.

**<<B>>** Secondary Data Sources

15
Other studies of the RHVP have been undertaken since the program was launched in 2009. To provide as complete a picture as possible in this evaluation report, the authors draw data and findings from these other sources.

In 2011 Mazzilli conducted a cost-modeling exercise for the HealthyBaby voucher service, adopting a Ugandan SP’s perspective of public or external resources. She drew on voucher sales and claims data, as well as cost data, which included reimbursements to voucher SPs for actual services offered to clients, along with program operation costs (Mazzilli 2011). These costs were obtained from the program headquarters in Mbarara and drawn from the program database, a review of financial reports, and interviews with program staff. The government facility costs were drawn from national sources (Mazzilli 2011).

Brody and colleagues conducted semistructured interviews on workload and job satisfaction in a total of 35 facilities, including eight that accepted the HealthyBaby vouchers, seven that accepted the HealthyLife vouchers, another seven that accepted both vouchers, and 13 nonvoucher facilities (Brody, Irige, and Bellows 2015). In each of the facilities, SPs were selected for interviews based on the following criteria: (1) proprietors and administrators, (2) clinical officers, midwives, and laboratory technicians, and (3) nurses, nursing assistants, and laboratory assistants. A total of 76 SPs were interviewed, with 56 from the intervention facilities and 20 from comparison facilities (Brody, Irige, and Bellows 2015). Job satisfaction was measured using an adapted version of the ten-item Minnesota Job Satisfaction Scale.

# **<<A>>** Analysis

The evaluation of the population-level impact of the voucher program was initially designed to approximate the counterfactual—that is, what the outcome of the program would have been in the absence of OBA voucher subsidies for comparison with the outcome of the actual interventions. However, since the design was not implemented as planned, the analytical approach adopts a post hoc definition of the counterfactual. Two definitions are used: (1) respondents in the 2010–11 survey who had never used the maternal health voucher (nonvoucher clients), and (2) respondents from villages that were included in both baseline and follow-up surveys, but where no voucher clients were present in 2010–11.

The outcomes of interest include targeting of beneficiaries, knowledge, health service access and utilization (new use and general use of ANC, health facility delivery, and PNC), outof-pocket expenses, and socioeconomic inequities in service utilization. An analysis of targeting 16
of beneficiaries entailed cross-tabulation of the indicator to determine whether the respondent had ever used the maternal health voucher by household wealth index among women who participated in the 2010–11 survey. Analysis of health service utilization that considered births occurring before and after the voucher program was initiated among voucher clients (women who had used the maternal health voucher) and nonvoucher clients. The impact of the program on health service utilization was determined by the difference-in-differences estimate—that is, the difference in changes over time between voucher and nonvoucher clients (Gertler et al. 2010).

The difference-in-differences estimate was obtained both from a simple comparison of changes in proportions utilizing services as well as from an estimation of multilevel, randomintercept, logit models, due to the hierarchical nature of the data. The multilevel logit models include an interaction term between the indicator of whether a voucher or nonvoucher client gave birth and the period of occurrence (birth before or after the program started). The models controlled for maternal age at birth, education level, marital status, place and duration of residence, religious affiliation, poverty status, parity, birth order, and sex of child.

The impact of the program on out-of-pocket expenses involved a simple difference-indifferences comparison of changes in the proportions paying for delivery, and an estimation of multilevel, random-intercept, logit models predicting the likelihood of paying for delivery at any public or private facility. With respect to equity, the analysis examined the gap in the use of maternal health services by poor and nonpoor women in villages with and without a voucher client in the five years preceding the 2010–11 survey. The impact of the program on reducing inequity was examined by performing a simple comparison of the difference in the proportions of poor and nonpoor women using the services in villages with and without a voucher client, and an estimation of multilevel random-intercept models.

The multilevel logit models included interaction terms between the survey year and whether a voucher client was resident in the village at follow-up. The models control for maternal age at birth of the most recent child, level of education, marital status, place and duration of residence, religious affiliation, poverty status, and the number of children born at any time (parity). The basic form of the multilevel, random-intercept, logit model with interaction terms is given by Equation [1]: 17
where _X1_ is the indicator for the period of birth occurrence, _X2_ is the indicator for exposure to the voucher program, and _Xi_ is the vector of the control variables in the model for birth _i_ from village _j_ in parish _k_ . The parameter _α0_ represents the likelihood of the outcome for nonexposed individuals at baseline, _α1_ is the difference in the outcomes for nonexposed individuals over time, _α2_ is the difference in outcomes between exposed and nonexposed individuals at baseline, _α3_ is the difference in the changes in outcomes between exposed and nonexposed individuals over time (i.e., the difference-in-differences estimate); _βi_ is the vector of parameters for the control variables in the model, and  _jk_ are the unobserved characteristics of individuals from the same village and parish that might be correlated with the outcome.

A key assumption of the difference-in-differences estimation is that preexisting trends between intervention and comparison groups are similar. Table 1 presents trends in the distribution of facility-based births occurring two years before the voucher program began among multiparous voucher and nonvoucher respondents who were interviewed in the 2010–11 survey. The results show no significant baseline differences between the two groups of respondents in the proportion of births delivered in public or private facilities in the two years preceding the voucher program launch.

Table 1 Trends in Facility-Based Births among Multiparous Voucher and Nonvoucher Respondents for Births Occurring Two Years before the Voucher Program, 2010–11

||Public|health facility||Privat|e health facility||
|---|---|---|---|---|---|---|
|Year of<br>birth|Voucher<br>respondents|Nonvoucher<br>respondents|_P_<br>value|Voucher<br>respondents|Nonvoucher<br>respondents|_P_<br>value|
|2006|50.0%|51.7%|0.83|<br>20.0%|12.8%|0.20|
||(_N_= 50)|(_N_= 180)||(_N_= 50)|(_N_= 180)||
|2007|43.9%|38.9%|0.46|<br>24.2%|21.1%|0.59|
||(_N_= 66)|(_N_= 247)||(_N_= 66)|(_N_= 247)||

_Source:_ Population Council, Household Survey 2010–11.

_Note: P_ values are from significance tests of proportions for differences between voucher and nonvoucher respondents.

18
# **<<B>>** Descriptive Statistics

Analysis of the 2010 HFA and other data sets involves simple descriptive statistics (percentages for categorical outcomes). Where data permit, a cross-sectional comparison is made between voucher and nonvoucher sites. Significance tests of proportions are performed to determine whether individuals from voucher and nonvoucher sites are significantly different with respect to the outcomes of interest.

# **<<A>>** Results

The following section describes aspects of program performance and study outcomes including the uptake of different voucher services among the targeted beneficiaries, reproductive health knowledge among the study respondents, voucher-subsidized health care use among the study respondents; costs of service and program management, survey respondents’ out-of-pocket costs; poverty-targeting effectiveness, and differences in quality of care at facilities in the control and intervention study arms.

# **<<B>>** Targeting Beneficiaries

Voucher sales data show that a total of 102,562 HealthyBaby vouchers were sold during the project period (2008–12). Of these, 65,590 (64 percent) were used for SD; 86,894 (85 percent) for one ANC visit; 18,131 (18 percent) for PNC; 8,887 (9 percent) for pregnancy complications; and 4,013 (4 percent) for emergency transport (figure 1). The number of vouchers used for safe deliveries represented 130 percent of the targeted 50,456 deliveries. The annual number of deliveries subsidized by the program increased from 2,837 in 2009 to 28,193 in 2010 and to 35,442 in 2011, reflecting a growing demand for the vouchers. Of the total 65,590 deliveries, 9,044 were by Caesarean section (13.8 percent), which falls within the World Health Organization–recommended rate of between 5 and 15 percent. The impact of the project was exemplified in Kasese district, where seven SPs out of a total of 36 health facilities contributed more than 50 percent of all estimated deliveries in the district.

The program started in the final quarter of 2008, but it is worth noting that there is always a lag between voucher sales and redemption, affecting the rate of increase in the number of subsidized deliveries. Another reason for the gradualness of the increase was the time required to accredit and contract facilities, identify voucher distributors, and conduct other preparatory activities.

19
index shows that a significantly higher proportion of voucher respondents compared with nonvoucher respondents were from the poorest 40 percent of households (52 percent vs. 37 percent; _P_ <0.01; see table 2). No analysis was done to place voucher beneficiaries into a national poverty scale. The wealth quintiles reported here refer only to the study population.

Table 2 Percentage Distribution of HealthyBaby Voucher and Nonvoucher Respondents, by Household Wealth Index

|Household wealth index|Voucher<br>respondents|Nonvoucher<br>respondents|All women|
|---|---|---|---|
|Poorest quintile|28.3|18.8|20.8|
|Poorer quintile|23.3|17.9|19.1|
|Middle quintile|14.5|20.8|19.4|
|Richer quintile|18.1|20.7|20.1|
|Richest quintile|15.7|21.9|20.5|
|Number of respondents|502|1,811|2,313|

_Source:_ Population Council, Household Survey 2010–11.

_Note:_ Percentages may not sum to 100 in some cases because of rounding.

**<<B>>** Knowledge of Maternal Health Services and Conditions

Awareness of the HealthyBaby voucher increased over time. In particular, among women who were interviewed in the 2008 household survey, 21 percent had heard about the HealthyBaby voucher, which was promoted on radio in late 2008 before voucher sales commenced and well before the first delivery using the voucher in late January 2009. This more than tripled to 76 percent among women interviewed in the 2010–11 survey. The proportion of women who were from villages included in both the 2008 and 2010–11 surveys and who had heard about the HealthyBaby voucher also increased from 22 percent in 2008 to 75 percent in 2010–11. In addition, 78 percent of those living in villages with a voucher user in 2010–11 had heard about the HealthyBaby voucher, compared with 69 percent of those from villages with no voucher user. Results from the household surveys further show that women who had heard about the HealthyBaby voucher were more likely to be aware of possible dangers during pregnancy, childbirth, or immediately after 21
childbirth compared with those who had not heard about the voucher (69 percent and 57 percent respectively), suggesting that the program not only created awareness about the vouchers but other reproductive health indicators as well. However, no significant difference was seen in knowledge of maternal health conditions between the health SPs from HealthyBaby voucher facilities who were interviewed in 2010–11 and those from nonvoucher sites (table 3).

Table 3 Percentage of Providers Interviewed in 2010–11 by Knowledge of Selected Maternal Health Conditions

|Maternal condition|HealthyBaby<br>voucher<br>facilities<br>(_N_= 63)<br>(%)|Comparison<br>facilities<br>(_N_= 49)<br>(%)|_P_value|
|---|---|---|---|
|Birth<br>preparedness<br>and<br>complications<br>readiness<sup>a</sup>|78|84|0.418|
|Danger signs in pregnancy|89|86|0.694|
|Danger signs during labor, delivery, and<br>afterward|86|89|0.659|
|Number of ANC visits women should make<br>during pregnancy|75|80|0.198|
|Gestational age at which a woman should make<br>first ANC visit|52|53|0.668|

_Source:_ Population Council, Health Facility Survey 2010–11.

_Note:_ ANC= antenatal care; _N_ = number of SPs interviewed.

a This statement was read to the respondents so they could indicate their level of knowledge regarding it.

# **<<B>>** Health Service Utilization

The net increase in private facility deliveries among women who had used the HealthyBaby voucher was greater than the reduction in public facility deliveries or home births (table 4). The proportion of private facility births among women who had ever used the HealthyBaby voucher in 22
the 2010–11 survey increased from 26 percent during the period before the voucher program to 52 percent after the program launched (table 4). Although an increase in private facility births also occurred among women who had never used the voucher, the increase was modest—from 18 percent before the program began to 28 percent after the program began—representing an increase of 10 percentage points. This difference-in-differences estimate of 16 percentage points between voucher and nonvoucher users is statistically significant, with an odds ratio of 2.2 (1.3–3.8). Further, results from the multilevel logit model show that reductions in public health facility delivery and home-based births were significantly higher among voucher respondents compared with nonvoucher respondents. The significant increase in private facility births and similar reductions in home-based births among voucher respondents compared with nonvoucher respondents are further indications that the program improved access to facility delivery. However, no significant difference was seen in changes in the use of PNC, or attending four or more ANC visits between voucher and nonvoucher respondents.

Table 4 Percentage of Births to Voucher and Nonvoucher Respondents by Use of Maternal Health Services and Difference-in-Differences Estimates in Changes over Time between Voucher and Nonvoucher Clients 2010–11

|Indicator of<br>health service<br>utilization|Vou<br>respon<br>(%|cher<br>dents<br>)|Nonv<br>respo<br>(|oucher<br>ndents<br>%)|Differenc<br>differen|e-in-<br>ces|
|---|---|---|---|---|---|---|
||Before<br>program|After<br>program|Before<br>program|After<br>program|Percentage<br>points<sup>a</sup>|Odds<br>ratios<sup>b</sup>|
|Four or more|55|70|49|56|8|1.4|
|ANC visits|(_N_=|(_N_=|(_N_=|(_N_=||(0.9–|
||183)|459)|779)|1,281)||2.2)|
|Place of|(_N_=|(_N_=|(_N_=|(_N_=|||
|delivery|175)|434)|708)|1,184)|||
|Home|30|17|38|31|−6|0.6*<br>(0.3–<br>0.9)|

23
|Private<br>facility|26|52|18|28|16|2.2**<br>(1.3–<br>3.8)|
|---|---|---|---|---|---|---|
|Public<br>facility|44|30|43|41|12|0.5*<br>(0.3–<br>0.9)|
|Public or|70|82|61|69|4|1.6|
|private||||||(0.9–|
|facility||||||2.8)|
|PNC services|60|67|45|53|−1|1.1|
||(_N_=|(_N_=|(_N_=|(_N_=||(0.7–|
||183)|459)|779)|1281)||1.8)|

_Source:_ Population Council, Endline Household Survey 2011 a Based on differences in changes in proportions using health services: negative sign means the change was greater in the comparison group.

b Based on multilevel logit models with interaction terms; 95 percent confidence intervals are in parentheses (* _P_ <0.05; ** _P_ <0.01).

# **<<B>>** Service, Program, and Out-of-Pocket Costs

In terms of service costs, the average reimbursement for a normal delivery rose from $14 in the 2008–09 financial year to $15 in 2009–11 and $19 in 2010–11 (figure 2; Mazzilli 2011). The corresponding figures for a complicated delivery were $61, $80, and $88, respectively. The transition from an individually negotiated reimbursement fee to a program-wide fee schedule partly contributed to the rise in the average reimbursements between the 2009–10 and 2010–11 financial years (Mazzilli 2011). The average reimbursement cost for a normal delivery over the three-year period (2008–11) was $18, whereas that of a complicated delivery was $87. Reimbursement costs declined between mid-2010 and early 2011 (between arrows 2 and 3 in figure 3) due to the suspension of voucher sales (Mazzilli 2011).

24
Figure 2 Average Reimbursement for Facility Delivery 2008–11 <!-- Start of picture text --> Amount ($)<br><!-- End of picture text --> <!-- Start of picture text --> Complicated<br>Complicat edd elivery,<br>delivery, 2010 – 11,<br>2009 – 10, 88.29<br>Complicated<br>79.54<br>delivery,<br>2008 – 09,<br>61.89<br>Normal<br>Normal<br>Normal<br>delivery,<br>2008delivery, – 09, 2009delivery, – 10, 201019.09 – 11,<br>15.49<br>13.58<br>Normal delivery Complicated delivery<br><!-- End of picture text --> <!-- Start of picture text --> ––<br><!-- End of picture text --> _Source:_ Mazzilli 2011.

The program management cost of the HealthyBaby voucher service was initially higher than that of service reimbursement to health facility deliveries.<sup>1</sup> However, as sales of vouchers increased (indicated by arrow 1 in figure 3), program costs declined and reimbursement costs increased. Reimbursement costs declined between mid-2010 and early 2011 (between arrows 2 and 3) because of the suspension of voucher sales (Mazzilli 2011).

Figure 3 HealthyBaby Voucher Reimbursement versus Program Costs 2008–11

> 1 According to the Cost Analysis Report for the Reproductive Health Voucher Pilot Project in Western Uganda, Report for the World Bank and the Ministry of Health of Uganda, May 2013, the total project expenditures were as follows: 63% as direct reimbursements for service delivery; 29% as Programme Management Overheads, including salaries of all staff; and 8% on BCC and voucher management related costs like production, distribution, and claims processing.

25
<!-- Start of picture text --> $100,000 ==Delivery<br>$90,000 reimbursements<br>==Programme costs i<br>$50,000 (central)<br>$70,000<br>$60,000<br>$90,000<br>$40,000<br>$30,000 ick<br>$20,000<br>$10,000<br>$0<br>Q4 Qt @ 8 24 Qi @ 8 24 QI<br>2008 2009 2009 2009 2009 2010 2010 2010 2010 2011 2vti<br><!-- End of picture text -->
With respect to out-of-pocket spending on services, among villages included in the population surveys with a voucher client who delivered most recently at a private facility, the proportion of respondents who paid for the services declined by 44 percentage points between 2008 and 2010–11 (table 5). In villages with no voucher client at follow-up, the proportion of women who delivered their most recent babies at a private facility and paid for the services declined by only 11 percentage points between 2008 and 2010–11. The decline in the proportion of women paying for delivery in private facilities was therefore greater—by 33 percentage points—in villages with a voucher client than in villages with no voucher client at follow-up. The difference-in-differences estimate from the multilevel logit model shows that the decline was significantly greater in villages with a voucher client, compared with villages with no voucher client.

Table 5 Percentage of Women Who Paid for the Delivery of the Most Recent Birth and Difference-in-Differences Estimates in Changes over Time between Villages with and Villages without a Voucher Client Resident in 2010–11

|Services|Vouch<br>resident<br>at fol<br>(|er client<br>in village<br>low-up<br>%)|No vou<br>resident<br>at fol<br>(|cher client<br>in village<br>low-up<br>%)|Differen<br>differe|ce-in-<br>nces|
|---|---|---|---|---|---|---|
||2008|2010–11|2008|2010–11|Percentage<br>points<sup>a</sup>|Odds<br>ratios<sup>b</sup>|
|Paid-for delivery<br>at|||||||
|Private facility|98|54|97|86|33|0.1*|
||(_N_=|(_N_=|(_N_=|(_N_= 21)||(0.0–|
||206)|133)|112)|||0.9)|
|Public facility|30|25|24|13|−6|2.2|
||(_N_=|(_N_=|(_N_=|(_N_= 60)||(0.7–|
||327)|149)|180)|||6.9)|

27
|Public or|56|39|52|32|−3|0.9|
|---|---|---|---|---|---|---|
|private facility|(_N_=|(_N_=|(_N_=|(_N_= 81)||(0.4–|
||533)|282)|292)|||2.1)|

_<mark>Source:</mark>_ <mark>Population Council, Household Survey 2010–11</mark> a Based on differences in changes in proportions that paid for delivery services: negative sign means the change was greater in the comparison group.

b Based on multilevel logit models with interaction terms; 95 percent confidence intervals are in parentheses (* _P_ <0.05).

The proportion of women who paid for delivery of the most recent birth in any health facility (public or private) declined by 17 percentage points in villages with a voucher client and by 20 percentage points in villages with no voucher client at follow-up between 2008 and 2010– 11 (table 5). The difference-in-differences estimate from the multilevel logit model was not statistically significant.

# **<<B>>** Socioeconomic Inequities

A difference of 11 percentage points was seen between births to poor mothers and births to nonpoor mothers who made four or more ANC visits in villages with a voucher client at follow-up (table 6). In villages with no voucher client, a difference of 15 percentage points was seen. Although the gap was greater between poor and nonpoor women in villages without a voucher client compared with villages with such a client, the difference-in-differences estimate from the multilevel logit model was not statistically significant.

The results further show that the proportion of births to poor women who were delivered at a private facility was greater in villages with a voucher client compared with villages with no voucher client (table 6). The inclusion of wealth quintiles in the multivariate model was one attempt to control for different distributions in poverty levels in voucher and nonvoucher villages. Nonetheless, the difference in the proportion of births delivered by poor and nonpoor women at a private facility was greater by 7 percentage points in villages with a voucher client compared with villages with no voucher client. The difference-in-differences estimate from the multilevel logit model was, however, not statistically significant. A similar pattern was noted for births to mothers who received PNC.

28
Table 6 Percentage of Births and Difference-in-Differences Estimates in Utilization by Poverty Status between Villages with and Villages without a Voucher Client Resident in 2010–11

|Services|Vouch<br>resident<br>at fol<br>(|er client<br>in village<br>low-up<br>%)|No vouc<br>resident i<br>at foll<br>(|her client<br>n village<br>ow-up<br>%)|Differen<br>differe|ce-in-<br>nces|
|---|---|---|---|---|---|---|
||Poor|Nonpoor|<br>Poor|Nonpoor|Percentage<br>points<sup>a</sup>|Odds<br>ratios<sup>b</sup>|
|Four or more ANC<br>visits|51<br>(_N_=<br>397)|62<br>(_N_=<br>573)|37<br>(_N_= 82)|52<br>(_N_=<br>216)|−4|1.1<br>(0.6–<br>2.3)|
|Place of delivery|(_N_=<br>364)|(_N_=<br>517)|(_N_= 80)|(_N_=<br>198)|||
|Private facility|25|35|13|16|7|0.8<br>(0.3–<br>2.1)|
|Public facility|31|35|32|48|−12|1.5<br>(0.7–<br>3.3)|
|Public or<br>private<br>facility|59|74|45|69|−9|0.9<br>(0.5–<br>2.0)|
|PNC|45|59|51|52|13|0.5|
||(_N_=|(_N_=|(_N_= 82)|(_N_=||(0.3–|
||397)|573)||216)||1.1)|

_<mark>Source:</mark>_ <mark>Population Council, Household Survey 2010–11.</mark> a Based on differences in the proportions of poor and nonpoor women who used the health services in villages with and without a voucher client: a negative sign means the difference between the poor and nonpoor was greater in the comparison group.

29
b Based on multilevel logit models with interaction terms; 95 percent confidence intervals are in parentheses (* _P_ <0.05).

# **<<B>>** Quality of Care

Anecdotal evidence suggested that the SPs used income generated from the HealthyBaby voucher program to upgrade facilities and improve service provision. These improvements included the opening of a maternity ward, hiring of additional staff, improving equipment, and bringing volunteers onto the payroll. However, results from observations of CPIs (based on the quality-ofcare model of structure, process, and outcome of care) showed that although voucher facilities performed better in terms of adequately creating a rapport with clients (e.g., greeting clients and ensuring privacy and confidentiality) during ANC consultations compared with nonvoucher facilities, no significant difference was seen in the proportion of SPs that adequately took into account the general or medical history of clients. Similarly, no significant difference was seen in the proportion of SPs from voucher and nonvoucher facilities that adequately created a rapport or took into account the general history of PNC clients (table 7).

30
Table 7 Percentage of Providers Observed during Consultation from Voucher and Nonvoucher Facilities That Adequately Offered Various Services to Clients in 2010

|Indicator|Voucher facilities|Comparison facilities|
|---|---|---|
|ANC|(_N_= 96)|(_N_= 51)|
|Adequately created rapport<br>(three items)|79|67*|
|Adequately took account of<br>client’s general history (four<br>items)|26|26|
|Adequately took account of<br>client’s medical history (six<br>items)|18|10|
|PNC|(_N_= 59)|(_N_= 24)|
|Adequately created rapport<br>(four items)|20|13|
|Adequately took account of<br>client’s general history (four<br>items)|3|0|

_Source:_ Population Council, Health Facility Survey 2010–11.

_*_ Difference between voucher and nonvoucher facilities is statistically significant at 5 percent. For each indicator, percentages refer to the proportions of consultations at which the SP performed all the itemized tasks.

The results from the observations of CPIs further show that significantly higher proportions of SPs from voucher facilities conducted most of the clinical examinations for ANC and PNC clients compared with their counterparts at nonvoucher facilities (table 8).

31
Table 8 Percentage of Providers Observed during Consultation from Voucher and Nonvoucher Facilities That Performed Various Clinical Examinations in 2010

|Clinical examination|A|NC|P|NC|
|---|---|---|---|---|
||Voucher<br>facilities<br>(_N_= 95)|Comparison<br>facilities<br>(_N_= 51)|Voucher<br>facilities<br>(_N_= 59)|Comparison<br>facilities<br>(_N_= 24)|
|Weight|70|55|n.a.|n.a.|
|Height|21|4**|n.a.|n.a.|
|Temperature|19|6*|10|4|
|Blood pressure|80|51**|34|4**|
|Anemia check|66|59|39|4**|
|Breasts/nipples exam|n.a.|n.a.|20|0*|

_Source:_ Population Council, Health Facility Survey 2010–11.

_Note:_ Differences between voucher and nonvoucher facilities are significant at * _P_ <0.05, ** _P_ <0.01. Abbreviation n.a. = not applicable.

Among women who participated in the 2010–11 household survey, nearly all (98 percent) of those who had ever used the HealthyBaby voucher indicated they would recommend its use to a friend. Their most commonly cited reasons included the fact that (1) it offers free, cheap, or affordable services, (2) it is good for or helps poor pregnant women access relevant services, and (3) it enables women to receive good quality or prompt service. Among the 2 percent who indicated they would not recommend the use of a voucher to a friend, the major reasons were (1) poor quality services, including rude, untrained, or uncaring SPs; (2) lack of drugs and unavailability of some services; (3) double payment for services; and (4) distance to accredited facility.

In terms of workload, an independent study using semistructured interviews with SPs showed that nearly all (98 percent) of those from voucher facilities reported an increase in the number of clients during the year preceding the interview date, as compared with 30 percent of SPs from nonvoucher facilities (Brody et al. 2015). This, however, had varied consequences. On the one hand, some SPs reported increased revenue as a result of increased workload, which, in turn, enabled them to retain or hire more staff and improve facility infrastructure. Other SPs 32
reported gaining more experience and improving their skills as a result of the increased number of clients. On the other hand, 30 percent of the SPs complained of overwork from caring for two or three times more clients than they had previously cared for (Brody et al. 2015).

With respect to job satisfaction, no significant difference was seen in the mean job satisfaction score reported by SPs from voucher and nonvoucher facilities, suggesting that the staff at voucher facilities were not significantly overworked (mean score of 23.1 and 21.6, respectively; Brody, Irige, and Bellows 2015). However, among higher and midlevel SPs (proprietors, administrators, and departmental managers), job satisfaction was significantly higher at voucher facilities compared with nonvoucher facilities (Brody, Irige, and Bellows 2015). Moreover, in voucher facilities, job satisfaction was significantly higher among higher-level and midlevel SPs than among lower-level SPs such as nurses, nursing assistants, and laboratory assistants (Brody, Irige, and Bellows 2015).

Despite the many positive outcomes of the RHVP and the improvement in general—as expected—the sample size limited the ability to identify significant differences in changes in the gap between poor and nonpoor women making four or more ANC visits, delivering at a private facility, or receiving PNC among women in villages with a voucher client compared with villages with no voucher client. Further, no significant differences were seen between voucher and nonvoucher facilities with respect to certain clinical aspects of quality of care, perhaps because SPs lacked regular training or incentive to update skills. In addition, a number of challenges were identified by the IVEA. As of the end of 2010, the IVEA found challenges existed regarding regular training of SPs, quality of services in some facilities, compliance with the frequency and format of reporting by facilities, and administration of the audit trails generated by the claimsprocessing database management system (IVEA 2010). However, the VMA had addressed some of these challenges by the beginning of 2012, including improving the verification and claims processes and ensuring that contracted health facilities meet performance standards (IVEA 2012). In addition, given that there was no random assignment of sites to the voucher program, differences between voucher and nonvoucher sites could be due to unobserved differences in characteristics between the sites that could bias the estimated effect of the program.

# **<<A>>** Conclusion

One of the major findings of this study is that the voucher program accelerated a shift from home to facility deliveries among voucher clients, as compared with nonvoucher clients, and was 33
sufficiently large to have significantly contributed to population-level reductions in the likelihood of paying out-of-pocket for deliveries in private health facilities. In particular, significant reductions were seen between 2008 and 2010–11 in the likelihood of paying out-of-pocket for private facility births in villages with a resident voucher client, as opposed to villages with no voucher client in residence. The voucher program also led to improvements in certain aspects of quality of care, such as the opening of a maternity ward, hiring of additional staff, and upgrading equipment.

The second major finding of the evaluation is that a significantly higher proportion of women from the poorest households used the HealthyBaby vouchers, compared with those from better-off households. This is an indication of the program’s success in targeting the poor. Nonetheless, a fraction of women from the better-off households also used the vouchers, suggesting the need for strengthening the targeting mechanisms. The finding might, however, be influenced by the measure of poverty used—that is, the household wealth index rather than the actual criterion used by the VMA to identify beneficiaries. Given the imperfect overlap in measurement, the household wealth index could either include or exclude women who would have qualified based on the VMA poverty grading tool used to identify beneficiaries. However, the VMA poverty grading tool was not administered to survey respondents. Had this been done, it could have helped determine the extent to which the two measures overlap.

To accelerate progress toward UHC in Uganda, much greater service uptake among the disadvantaged and poor is needed as well as substantial improvements in quality of care. Voucher programs could accelerate uptake of maternal health care among “new users,” who would not otherwise have sought care. Voucher reimbursements to facilities have the potential to improve quality of care, particularly in infrastructure and facility physical stock. However, the program must continue to innovate and seek efficiency gains as more facilities are contracted and the volume of clients expands.

It is worth considering how the maternal health care voucher is nested within the health system. There are approximately 1.3 million births a year in Uganda and, among the bottom 40 percent of the population, a significant majority of the deliveries takes place at home, far from any skilled attendant or emergency obstetric care. If the voucher program is scaled nationally, it must address demand and supply side issues. On the demand side, its focus ought to be on reaching the poor “new user” who has a high probability of a home birth. On the supply side, the program ought 34
to find an appropriate mechanism to contract public facilities to improve the geographic coverage and direct reimbursements to public facilities that are serving voucher beneficiaries.

The voucher strategy serves as an instrument to cover poor, informal-sector households with a minimum package that is time delimited and less expensive than a full insurance package (a key recommendation in a recent report: Bitran 2014). Covering the informal sector is a challenge to policy makers who want to ensure disadvantaged populations are receiving the best available services while also avoiding costly packages that may not be affordable over the medium term. Depending on the content of the service package, voucher programs can serve as a compromise between providing a full, demand-side finance package of inpatient and outpatient services versus a supply-side approach that may neglect to prioritize services to the poorest of society.

35
# **<<A>>** References

Abuya, T., R. Njuki, C. E. Warren, J. Okal, F. Obare, L. Kanya, I. Askew, and B. W. Bellows. 2012. “A Policy Analysis of the Implementation of a Reproductive Health Vouchers Program in Kenya.” _BMC Public Health_ 12 (1): 540.

- Bellows, N. M., B. W. Bellows, and C. Warren. 2011. “The Use of Vouchers for Reproductive Health Services in Developing Countries: Systematic Review.” _Tropical Medicine and International Health_ 16 (1): 84–96.

- Bellows, B. W., C. M. Conlon,  E. S. Higgs, J. W. Townsend, M. G. Nahed, K. Cavanaugh, A. C. Gorter. 2013. “A Taxonomy and Results from a Comprehensive Review of 28 Maternal Health Voucher Programmes.” _Journal of Population, Health, Nutrition_ 31 (4): S108–S128.

- Bellows, B. W., C. Kyobutungi, M. K. Mutua, C. Warren, and A. Ezeh. 2013. “Increase in Facility-Based Deliveries Associated with a Maternal Health Voucher Programme in Informal Settlements in Nairobi, Kenya.” _Health Policy and Planning_ 28 (2): 134–42.

- Bhat, R., D. V. Mavalankar, P. V. Singh, and N. Singh. 2009. “Maternal Healthcare Financing: Gujarat’s Chiranjeevi Scheme and Its Beneficiaries.” _Journal of Health, Population, and Nutrition_ 27 (2): 249–58.

- Bhatia, M. R., & and A. C. Gorter. 2007. “Improving Access to Reproductive and Child Health Services in Developing Countries: Are Competitive Voucher Schemes an Option?” _Journal of International Development_ 19 (7): 975–81.

- Bitran, R. 2014. _Universal Health Coverage and the Challenge of Informal Employment: Lessons from Developing Countries_ . World Bank: Washington, DC.

- Borghi, J., A. C. Gorter, P. Sandiford, and Z. Segura. 2005. “The Cost-Effectiveness of a Competitive Voucher Scheme to Reduce Sexually Transmitted Infections in High-Risk Groups in Nicaragua.” _Health Policy Plan_ 20 (4): 222–31.

- Brody, C. D., Irige, J. M., & Bellows, B.  W. 2015. “Burnout at the frontline: The impact of a reproductive health voucher program on health worker satisfaction in Uganda.” _International Archives of Nursing and Health Care_ 1(1), 1–6.

- Cave, M. 2001. “Voucher Programmes and Their Role in Distributing Public Services.” _Organization for Economic Cooperation and Development Journal on Budgeting_ 1(1): 59– 88.

- Cernada, G., and L. Chow. 1969. “The Coupon System in an Ongoing Family Planning Program.” _American Journal of Public Health_ . 59(12): 2199–2208.

- Densmore, M. 2012a. “Experiences with Bulk SMS for Health Financing in Uganda.” Extended abstract in Proceedings of the 2012 SIGCHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery (ACM).

- Densmore, M. 2012b. “Claim Mobile: When to Fail a Technology.” Extended abstract in Proceedings of the 2012 SIGCHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery (ACM).

- Gertler, P. J., P. Premand, S. Martinez, C. M. J. Vermeersch, and L. B. Rawlings. 2010. _Impact Evaluation in Practice_ . World Bank, doi:10.1596/978-0-8213-8541-8.

- Gorter, A. C., P. Sandiford, Z. Rojas, and M. Salvetto. 2003. _Competitive Voucher Schemes for Health: Background Paper_ . Mexico: ICAS.

- Gwatkin, D. R., and A. Ergo. 2011. “Universal Health Coverage: Friend or Foe of Health Equity?” _Lancet_ 377 (9784): 2160–61, doi:10.1016/S0140-6736(10)62058-2.

- IMF. 2013. _IMF Report for Selected Countries and Subjects: Uganda Economic Figures_ . Accessed May 9, 2013,

http://www.imf.org/external/pubs/ft/weo/2013/01/weodata/weorept.aspx?pr.x=31&pr.y=16& 36
sy=2009&ey=2012&scsm=1&ssd=1&sort=country&ds=.&br=1&c=746&s=NGDPD,NGDP DPC,PPPGDP,PPPPC,LP&grp=0&a=

- IVEA. 2010. _Independent Verification and Evaluation Report for the January 1, 2009 to June 30, 2010 Period: Final Report._ Kampala, 29.

- Kanya, L., F. Obare, C. Warren, T. Abuya, I. Askew, and B. W. Bellows. 2013. “Safe Motherhood Voucher Programme Coverage of Health Facility Deliveries among Poor Women in South-Western Uganda.” _Health Policy and Planning,_ <u>1–8,</u> doi:10.1093/heapol/czt079.

- Mavalankar, D. V, Vora, K. S., Ramani, K. V, Raman, P., Sharma, B., & Upadhyaya, M. (2009). Maternal health in Gujarat, India: a case study. Journal of Health, Population, and Nutrition, 27(2), 235–48.

- Mazzilli, C. 2011. “The Cost-Effectiveness of a Voucher Programme for Facility-Based Delivery in Six Districts of Western Uganda: A Critical Analysis.” Unpublished paper. London: Marie Stopes International, 68.

- Meuwissen, L. E., Gorter, A. C., Kester, A. D. M., & Knottnerus, J. A. (2006a). Can a comprehensive voucher programme prompt changes in doctors’ knowledge, attitudes and practices related to sexual and reproductive health care for adolescents? A case study from Latin America. Tropical Medicine & International Health, 11(6), 889–98. doi:10.1111/j.1365-3156.2006.01632.x

- Meuwissen, L. E., Gorter, A. C., Kester, A. D. M., & Knottnerus, J. A. (2006b). Does a competitive voucher program for adolescents improve the quality of reproductive health care? A simulated patient study in Nicaragua. BMC Public Health, 6, 204. doi:10.1186/14712458-6-204

- Meuwissen, L. E. LE, Gorter, A. C. A., & Knottnerus, A. A. J. (2006). Impact of accessible sexual and reproductive health care on poor and underserved adolescents in Managua, Nicaragua: a quasi-experimental intervention study. The Journal of Adolescent Health, 38(1), 56. doi:10.1016/j.jadohealth.2005.01.009

- Mohanan, M., Bauhoff, S., Forgia, L., Babiarz, S., Miller, G., Forgia, G. La, & Singer, K. (2014). Effect of Chiranjeevi Yojana on institutional deliveries and neonatal and maternal outcomes in Gujarat, India: A difference-in-differences analysis. Bulletin of the World Health Organization, (October 2013), 187–194.

- doi:http://dx.doi.org/10.2471/BLT.13.124644

- Musgrove, P. (2011). Financial and Other Rewards for Good Performance or Results: A Guided Tour of Concepts and Terms and a Short Glossary. Washington, DC.

- Obare, F., Warren, C., Abuya, T., Askew, I., & Bellows, B. (2014). Assessing the populationlevel impact of vouchers on access to health facility delivery for women in Kenya. Social Science & Medicine, 102, 183–189. Retrieved from

- http://www.sciencedirect.com/science/article/pii/S0277953613006825

- PS Consulting. 2009. _Introducing Voucher Systems for Healthcare Provision: Health Provider Baseline Survey._ <u>Kampala.</u>

- Ross, J., D. W. Han, S. Keeny, G. P. Cernada, T. C. Hsu, and T. H. Sun. 1970. “Korea/Taiwan 1969: Report on National Family Planning Programs.” _Studies in Family Planning_ 1 (54): 1– 16.

- Tura, G., M. Fantahun, and A. Worku. 2013. “The Effect of Health Facility Delivery on Neonatal Mortality: Systematic Review and Meta-Analysis. _BMC Pregnancy and Childbirth_ 13 (1): 18, doi:10.1186/1471-2393-13-18.

- Uganda DHS. 2011. _Uganda Demographic and Health Survey: Preliminary Report_ .

37
UNDP. 2013. “UNDP: International Human Development Indicators for Uganda.” Accessed May 9, 2013, http://hdrstats.undp.org/en/countries/profiles/UGA.html.

38