metadata: authors: 0: author: Di Giorgio,Laura
1: author: Evans,David
2: author: Lindelow,Magnus
3: author: Nguyen,Son Nam
4: author: Svensson,Jakob
5: author: Wane,Waly
6: author: Tarneberg,Anna Welander
count: Kenya,Madagascar,Mozambique,Niger,Nigeria,Senegal,Sierra Leone,Tanzania,Togo,Uganda
docna: 0: docna: An Analysis of Clinical Knowledge,
            Absenteeism, and Availability of Resources for Maternal and
            Child Health : A Cross-Sectional Quality of Care Study in 10
            African Countries
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Health Service Management and Delivery,Malaria,Leprosy,Communicable Diseases,Cholera,Tuberculosis,Pharmaceuticals Industry,Pharmaceuticals & Pharmacoeconomics
teratopic: Industry,Health, Nutrition and Population
abstracts: cdata!: This paper assesses the quality of
            health care across African countries based on health
            providers' clinical knowledge, their clinic attendance,
            and drug availability, with a focus on seven conditions
            accounting for a large share of child and maternal
            mortality: malaria, tuberculosis, diarrhea, pneumonia,
            diabetes, neonatal asphyxia, and postpartum hemorrhage. With
            nationally representative, cross-sectional data from 10
            countries in Sub-Saharan Africa, collected using clinical
            vignettes, unannounced visits, and visual inspections of
            facilities, this study assesses whether health providers are
            available and have sufficient knowledge and means to
            diagnose and treat patients suffering from common conditions
            amenable to primary health care. The study draws on data
            from 8,061 primary and secondary care facilities in Kenya,
            Madagascar, Mozambique, Nigeria, Niger, Senegal, Sierra
            Leone, Tanzania, Togo, and Uganda, and 22,746 health
            workers. These data were gathered under the Service Delivery
            Indicators program. Across all conditions and countries,
            health care providers were able to correctly diagnose 64
            percent of the clinical vignette cases, and in 45 percent of
            the cases, the treatment plan was aligned with the correct
            diagnosis. For diarrhea and pneumonia, two common causes of
            under-five deaths, 27 percent of the providers correctly
            diagnosed and prescribed the appropriate treatment for both
            conditions. On average, 70 percent of health workers were
            present in the facilities to provide care during facility
            hours when those workers were scheduled to be on duty. Taken
            together, the estimated likelihood that a facility has at
            least one staff present with competency and the key inputs
            required to provide child, neonatal, and maternity care that
            meets minimum quality standards is 14 percent. Poor clinical
            knowledge is a greater constraint in care readiness than
            drug availability or health workers' absenteeism in the
            10 countries. However, the paper documents substantial
            heterogeneity across countries.
geo_reg_key: 1927320,517195,517191
repnme: repnme: An Analysis of Clinical Knowledge,
            Absenteeism, and Availability of Resources for Maternal and
            Child Health : A Cross-Sectional Quality of Care Study in 10
            African Countries
display_title: An Analysis of Clinical Knowledge,
            Absenteeism, and Availability of Resources for Maternal and
            Child Health : A Cross-Sectional Quality of Care Study in 10
            African Countries
listing_relative_url: /research/2020/10/32489636/analysis-clinical-knowledge-absenteeism-availability-resources-maternal-child-health-cross-sectional-quality-care-study-10-african-countries
topicv3: Tuberculosis m1326131 1419,Health Workforce m1326573 631,Malaria m1327042 1420,Pharmaceuticals m1327740 1350,Health m1504661 3480
publishtoextweb_dt: 2020-10-15T00:00:00Z
historic_topic: Industry,Health, Nutrition and Population
origu: HEALTH AFR E2 (HAEH2)
no_of_pages: 69
author_keys: 0: author_key: 1594960
1: author_key: 588541
2: author_key: 166917
3: author_key: 125655
4: author_key: 441050
model_extractions: Policy Research Working Paper 9440

# An Analysis of Clinical Knowledge, Absenteeism, and Availability of Resources for Maternal and Child Health

A Cross-Sectional Quality of Care Study in 10 African Countries _Laura Di Giorgio David K. Evans Magnus Lindelow Son Nam Nguyen Jakob Svensson Waly Wane Anna Welander Tärneberg_ Health, Nutrition and Population Global Practice October 2020
Policy Research Working Paper 9440

# **Abstract**

This paper assesses the quality of health care across African countries based on health providers’ clinical knowledge, their clinic attendance, and drug availability, with a focus on seven conditions accounting for a large share of child and maternal mortality: malaria, tuberculosis, diarrhea, pneumonia, diabetes, neonatal asphyxia, and postpartum hemorrhage. With nationally representative, cross-sectional data from 10 countries in Sub-Saharan Africa, collected using clinical vignettes, unannounced visits, and visual inspections of facilities, this study assesses whether health providers are available and have sufficient knowledge and means to diagnose and treat patients suffering from common conditions amenable to primary health care. The study draws on data from 8,061 primary and secondary care facilities in Kenya, Madagascar, Mozambique, Nigeria, Niger, Senegal, Sierra Leone, Tanzania, Togo, and Uganda, and 22,746 health workers. These data were gathered under the Service Delivery Indicators program. Across all conditions and countries, health care providers were able to correctly diagnose 64 percent of the clinical vignette cases, and in 45 percent of the cases, the treatment plan was aligned with the correct diagnosis. For diarrhea and pneumonia, two common causes of under-five deaths, 27 percent of the providers correctly diagnosed and prescribed the appropriate treatment for both conditions. On average, 70 percent of health workers were present in the facilities to provide care during facility hours when those workers were scheduled to be on duty. Taken together, the estimated likelihood that a facility has at least one staff present with competency and the key inputs required to provide child, neonatal, and maternity care that meets minimum quality standards is 14 percent. Poor clinical knowledge is a greater constraint in care readiness than drug availability or health workers’ absenteeism in the 10 countries. However, the paper documents substantial heterogeneity across countries.

This paper is a product of the Health, Nutrition and Population Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at ldigiorgio@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **An Analysis of Clinical Knowledge, Absenteeism, and Availability of Resources for Maternal and Child Health: A Cross-Sectional Quality of Care Study in 10 African Countries**

Laura Di Giorgio<sup>1</sup> , David K. Evans<sup>2</sup> , Magnus Lindelow<sup>1</sup> , Son Nam Nguyen<sup>1</sup> , Jakob Svensson<sup>3</sup> , Waly Wane<sup>1</sup> , and Anna Welander Tärneberg<sup>4</sup> _This paper is forthcoming in the journal BMJ Global Health._ JEL codes: I15, I18, O15 Keywords: health, service delivery, quality of care, Sub-Saharan Africa

- 1 World Bank, Washington DC, USA.

- 2 Center for Global Development, Washington, DC, USA.

- 3 Institute for International Economic Studies, Stockholm University, Stockholm, Sweden.

- 4 Center for Economic Demography and Department of Economic History, Lund University School of Economics and Management, Lund, Sweden.
# **What is already known?**

- Many country-specific studies have shown that low- and middle-income countries suffer severe shortages of facility inputs such as number of providers, drugs, equipment, and medical supplies.

- Individual country reports of Service Delivery Indicator surveys from Kenya, Niger, Senegal, Tanzania, and Uganda use summary statistics to show low rates of knowledge and high rates of absenteeism among health service providers.

# **What are the new findings?**

- This study offers a single metric that quantifies the relative contribution of staff knowledge, staff availability, and other facility inputs to the probability of care readiness which meets minimum quality standards.

- Our work shows severe gaps in care readiness in various countries in Sub-Saharan Africa, and health provider knowledge is a particularly severe constraint on the readiness of care across countries. Across the countries sampled and the evaluated medical conditions, which account for major loss of life across the continent, health care providers were able to correctly diagnose only 64% of cases, and the treatment plan was aligned with the correct diagnosis in 45% of cases.

# **What do the new findings imply?**

- While effective health care systems need to include a variety of components (knowledge, effort, equipment, and essential medicines), not all components merit the same initial priority.

- Our findings suggest that in the absence of provider knowledge, even improvements in other key areas such as effort, medication, and equipment cannot save patients’ lives.

- As a result, health systems will need to invest in better knowledge for their health providers along with other resources.

2
# **Background**

There is a growing consensus that poor quality care is a major constraint in further improving health outcomes in low- and middle-income countries (LMICs).<sup>1–3</sup> Based on the most recent estimates, 8.6 million people per year in LMICs die from conditions that could be treated by health systems, and a majority of these deaths are due to receipt of poor quality care.<sup>4</sup> Measuring quality at various levels of the health system—especially where the system fails the people in need—remains rare in LMICs. Existing indicators of structural quality, such as availability of medicine, equipment, and qualified health attendants, are not sufficient to adequately capture the quality of care being offered.<sup>1</sup> The lack of quality measurement can help to explain both why there is limited research on how to effectively transform low-quality health systems into high-quality ones, and why there are few such successful reforms in LMICs. Actionable and credible data on quality, if properly disseminated, can play an important role in holding country health systems accountable for effective and safe care and can form the basis for quality improvement.<sup>5</sup> Our paper reports on large-scale assessments of clinical knowledge, health provider absenteeism, and means to provide care in Sub-Saharan Africa. We use data collected through clinical vignettes, unannounced visits, and visual inspections of over 20,000 providers in over 8,000 facilities in 10 Sub-Saharan African countries, which together represent over 40 percent of the region’s total population. We focus on conditions that account for a large proportion of maternal and child mortality. (We include the STROBE checklist for cross-sectional, observational data analysis in Appendix Section S1.<sup>6</sup> ) We use these data to measure dimensions we would argue are essential for providing quality care. Clinical vignettes tell us what providers know. Unannounced visits to facilities tell us if providers are available at health facilities. Visual inspections of facilities provide information as to the means that health workers have at their disposal. To put this work in 3
context, the provider assessment falls within the framework set out by Miller in 1990, that a holistic assessment of health worker competence should assess their clinical knowledge, if they know how to apply that knowledge, if they can show how to apply that knowledge, and if they actually apply that knowledge in practice.<sup>7</sup> Our vignette data capture a combination of clinical knowledge and their knowledge of how to apply it. Other, complementary methods (specifically, standardized patient studies) demonstrate whether health workers can show how to apply their knowledge and whether they actually apply that knowledge in practice.<sup>8-10</sup> Nationally representative standardized patient data are unavailable in the settings and for the wide array of conditions we study, but limitations revealed in the first aspects of the health worker assessment (what they know and if they know how to apply it) and to one aspect of practice (absenteeism) potentially imply even further limitations on the latter part (whether they can and do actually apply knowledge in practice).

# **Methods**

# **_Data_**

We draw on data from the Service Delivery Indicators (SDI) program—an ongoing initiative to collect informative and standardized measures of what providers know, what they do, and what they have to work with. The SDI program—piloted in 2010-2011—grew out of concern about quality of services in health and education, especially in Sub-Saharan Africa, most clearly (and perhaps most damagingly) manifested at the facility level, in fast-expanding systems of primary health provision.<sup>11,12</sup> To date, the SDI program has collected data from 10 countries: Kenya (2018), Madagascar (2016), Mozambique (2014), Nigeria (2013), Niger (2017), Senegal (2010), Sierra Leone (2018), Tanzania (2016), Togo (2014), and Uganda (2013), and has since expanded to countries outside the Africa region.<sup>13-17</sup> While every context is different, these 10 countries serve as a reasonable proxy for much of Sub-Saharan Africa, 4
with an average income per capita, life expectancy, and infant mortality similar to those of the continent as a whole (Appendix Table S2).

# **_Sampling_**

All country data sets were representative at the national level except for Nigeria, which, mainly owing to security concerns, covered 12 of 37 states and was representative at the state level. The surveys used a multistage cluster-sampling design to ensure data were representative of rural and urban areas, private and public facilities, and of each facility type (e.g., primary care clinics versus secondary care hospitals).

The surveys collected a broad set of facility and provider specific information, using standardized instruments, with an approach that relied on clinical vignettes, direct physical verification of provider presence through unannounced visits, and visual inspections of the equipment and facilities. Every interview respondent was asked for their consent to participate in the survey. Sample instruments are publicly available.<sup>18–22</sup> In total, the sample includes information from 89,826 vignettes from 13,754 providers and absenteeism observations from 22,747 providers across 8,061 facilities in 10 countries. In each country, the sample size was selected in order to provide representative data for the country and for sub-groups (e.g., rural versus urban areas).

# **_Empirical methods_**

There are three essential conditions for health workers to provide quality care. First, providers have sufficient knowledge to diagnose and treat patients suffering from conditions that are amenable to care. Second, they are available to provide care when patients seek service. Third, providers have the means (e.g., equipment and drugs) to diagnose and treat patients. We operationalize the assessment of these necessary, but not sufficient, conditions for quality primary care using three methods. Diagnostic and treatment accuracy (knowledge) were 5
measured using clinical vignettes focused on seven common conditions associated with the main causes of child and maternal mortality in Sub-Saharan Africa: malaria, tuberculosis, diarrhea, pneumonia, diabetes, neonatal asphyxia and post-partum hemorrhage. All seven of these are major global health challenges, with three of them – diarrhea, neonatal asphyxia, and malaria – together causing more than 180,000,000 disability-adjusted life years (Appendix Table S3). Whether providers are available to provide care at regular opening hours was assessed by verifying their presence through unannounced visits at the health facilities. Availability of drugs and functionality of equipment was assessed by visual inspections of storage facilities and consulting rooms at the health facilities.

Clinical vignettes have been used extensively in the medical literature to assess clinical knowledge of health workers. Data from vignette studies are directly policy relevant as they are closely linked to the quality of medical training and quality improvement programs. Validation studies show that clinical vignettes, for scenarios such as those assessed as part of the SDI program, are an effective way to capture provider knowledge.<sup>23,24</sup> Although a health worker’s clinical vignette performance is not a measure of quality of care per se, it represents the highest level of quality care which s/he can deliver given his/her current knowledge, assuming that there are no constraints in drugs, equipment and motivations. (More details on the clinical vignettes are provided in Section S4 in the appendix.) For a measure of providers’ actual performance, studies often use standardized patients, in which actors are trained to portray a given medical condition, appear incognito at facilities for recommended treatment, and report on provider activity.

The clinical vignettes were adjusted to country-specific diagnostic and treatment protocols. The enumerators presented the provider with the patient’s symptoms and prompted the provider to take the patient’s history, suggest physical examination and diagnostic tests 6
that in principle could be conducted onsite, make a diagnosis, and prescribe a course of treatment and follow-up management plan.

For this study, we identified the lowest common denominators across the countryspecific protocols to define comparable criteria across countries for diagnostic and treatment accuracy for each of the conditions. This methodological choice implies that knowledge is assessed at a lower standard than more stringent World Health Organization (WHO) protocols. We discuss cross-country variation in country-specific protocols in Appendix S6. Based on the responses to each vignette, we generated two binary variables: diagnostic accuracy and treatment accuracy, which are the percentage of respondents who provided an accurate diagnosis and – separately – an accurate treatment for a given condition. Both outcomes should be viewed as measuring the minimum required knowledge to prescribe correct diagnostic and treatment services. (See Table S7 for information on the coding of the two knowledge scores.) These outcomes do not represent the ideal level of knowledge; for example, they do not account for co-morbidities.

To quantify whether providers are available to provide medical service when patients seek care, we followed a standard procedure in which enumerators visited each facility twice.<sup>25</sup> During the first visit, information on all staff working at the facility, by professional cadre, was collected. Provider availability does not, of course, fully capture provider effort, but it is a necessary condition for treating patients. (See Section S8 in the appendix for details on the selection of staff, drugs, and equipment for surveys.) Vignettes were administered to a random sample of doctors, medical assistants, nurses, and nurse-midwives (i.e., any staff that provide outpatient or maternity services) who were present in the facility on the day of the first, announced visit. The first visit also included a visual inspection of drugs and equipment. A few days after the first visit, enumerators made a second, unannounced visit to the facility and assessed the presence of a randomly pre-selected list of providers (or all 7
providers, if fewer than 10 providers were available). Only staff on duty that day were included in the absenteeism measure. Staff who were away from the facility due to outreach or fieldwork were counted as present. Country-level averages for each assessment were calculated using country-specific sampling weights. This gives us a mean of the outcome in question that is representative of the country population.

# **_Estimates of care readiness_**

Combining data from the clinical vignettes (to measure knowledge) with data from the unannounced visit (to measure the extent to which providers are present in the facility at regular opening hours), and visual inspections (to quantify the availability of drugs and equipment), we estimate the probability that a facility is ready to provide care that meets minimum quality standards. A facility is defined as being ready to provide care that meets minimum quality standards for a given condition if there is at least one provider present in the clinic at a given day who can correctly diagnose and treat that condition, and the facility has a minimum set of drugs to treat it. (See Section S9 in the appendix for further details of the calculation.) The probability of care readiness that does not meet minimum quality standards represents the sum of two important quality gaps: the gap between what health workers are supposed to know and what health workers actually know, and the gap between what health workers know and what health workers can do (either due to absenteeism or a lack of essential drugs). By varying the underlying parameters used to construct the measure, we can assess the relative importance of these two gaps. This research does not examine an additional important gap, between what workers can do and what they actually do in practice. We report three estimates for care readiness for each condition. Our main estimate is based solely on the survey data. We then consider a hypothetical scenario in which all health workers are present: we replace the survey-based estimate of absenteeism at the facility level for a type of worker 8
with 0, and re-estimate care readiness. We then repeat the same procedure, but instead of replacing the estimate for absenteeism at the facility level, we assume all facilities have the essential drugs for the condition(s) in question.

We present averages and 95% confidence intervals by first averaging all observations across facilities (or health workers) in each country (using country-specific facility weights to calculate a representative country average), and then averaging over the country means to calculate an overall average for the whole sample. (This is a suitable approach if we treat the country as the core unit of observation; i.e., we want to know how countries compare and how they perform as a group.) This procedure is equivalent to calculating a weighted average and its standard error across all observations in the sample, where the weight on observation in country is i = ∏ i , where i denotes the (facility) weight of observation in ii kk≠ii kk country jj and ωω FF is the sum of the facility weights in country × ff ff . ii kk rr rrkk jj FF = ∑ff kk

# **_Ethical considerations_**

Each of these surveys was carried out in collaboration with the Ministry of Health in the target country. Each interview respondent provided consent to participate in the survey and was made aware that they could desist from the survey at any point. The study involved no deception of participants. The current study involves secondary analysis of the Service Delivery Indicators data.

# **_Patient and public involvement_**

Neither patients nor the public were involved in the design of this study. However, each Service Delivery Indicators survey was discussed – both at design and results stage – at length with representatives of the country where the survey was carried out. This research paper presents the analysis of data across 10 countries.

# **_Role of the funding source_**

9
The funding source did not have any role in the design, data collection, data analysis, data interpretation, or writing of the report.

# **Results**

# **_Diagnostic accuracy_**

Across all conditions and countries, health care providers correctly diagnose 64% (95% CI 0.62-0.65) of the cases. The poorest diagnostic performance is for diabetes, with only 35% (95% CI 0.33-0.37) of the health workers providing a correct diagnosis (Figure 1A; Table 1). For the three main killer diseases for children in Sub-Saharan Africa, accounting for approximately one-third of all deaths among children under age 5 in the countries surveyed, average diagnostic accuracy is 57% (95% CI 0.54-0.59) for pneumonia, 59% (95% CI 0.560.61) for diarrhea and 85% (95% CI 0.82-0.89) for malaria.

There are large variations in diagnostic accuracy across conditions and across countries (Figure 1A, Table 1). For example, only 11% (95% CI 0.04-0.18) of the providers in Togo correctly diagnose neonatal asphyxia, compared to 86% (95% CI 0.85-0.87) of the providers in Kenya, and only 18% (95% CI 0.09-0.26) of the providers in Togo correctly diagnose diarrhea, compared to 95% (95% CI 0.93-0.97) of the providers in Sierra Leone. The within-country variation also shows different patterns (see Figure S10), varying from relatively good performance, albeit still low in absolute terms, across the whole distribution of providers in the country (Kenya), to countries with relatively poor performance across the whole distribution (Madagascar), and countries that span the breadth of diagnostic accuracy (Nigeria). Figure S11 provides additional information on the clustering of diagnostic performance. Overall, 32 percent of the providers manage to diagnose all or almost all of the conditions (no more than one condition not correctly diagnosed), while 27 percent of the providers diagnose less than half of the conditions. There are again large differences across 10
countries. Figure S11 divides the countries into two groups based on performance. In the higher performance group, 49 percent of the health workers diagnose all or almost all of the conditions, compared to 11 percent in the lower performing group.

# **_Accuracy in prescribing treatment_**

Diagnostic knowledge is associated with knowledge of correct treatment, but a correct diagnosis is not a guarantee for prescribing correct treatment. The providers prescribed the correct (minimum) treatment following a correct diagnosis in 72% of the cases. Across all conditions and countries, health care providers were able both to correctly diagnose and treat 45% (95% CI 0.43-0.46) of the cases. For diarrhea, malaria, and neonatal asphyxia a majority of the providers managed to do so (54% (95% CI 0.43-0.46); 70% (95% CI 0.66-0.73); and 55% respectively). Taking diarrhea and pneumonia together, two common causes of underfive deaths, 27% (95% CI 0.25-0.29) of the providers correctly diagnosed and prescribed the appropriate treatment for both conditions. For the other four conditions, the majority of providers either fail to diagnose the condition correctly or fail to provide the correct treatment (Figure 1B; Table 2).

The required treatments varied in complexity across conditions, from prescribing a single drug or treatment (artemisinin-based combination therapies – ACTs – for malaria, dehydration therapy for diarrhea, amoxicillin or similar antibiotic for pneumonia) to prescribing a combination of drugs for tuberculosis. There is substantive heterogeneity in providers’ ability to reach a correct treatment after correctly diagnosing the condition. As a result, the relative ranking in ability to diagnose differs from the relative ranking in diagnosing and treating patients. For some conditions, the key constraint appears to be at the diagnostic stage, as with diarrhea, where 59% of the providers diagnosed correctly but 91% of those who did so prescribed the correct treatment, and neonatal asphyxia, where 59% of the 11
providers diagnosed correctly but 93% of those who did so prescribed the correct treatment. For other conditions, the key constraint instead appears to be prescribing the correct treatment, as with tuberculosis, where 81% of the providers correctly diagnosed the condition but only 37% of these providers prescribed the correct treatment, and post-partum hemorrhage, where 71% of the providers correctly diagnosed the condition but only 45% of them prescribed the correct treatment.

Misdiagnosis or poor knowledge about correct treatment can result in inappropriate prescription of antibiotics. The vignette for diarrhea does not call for prescription of antibiotics. However, prescription of antibiotics for diarrhea is common among providers both when the providers diagnosed the condition correctly—32% (95% CI 0.29-0.35) of the providers then prescribed antibiotics—and when they did not—36% (95% CI 0.32-0.40) of the providers then prescribe antibiotics (Figure 2; Table S12).

# **_Provider absenteeism_**

For quality care to be provided, providers must have sufficient knowledge to diagnose and treat patients. But providers must also be available. The rate of provider absence is, on average, 30% (95% CI 0.29-0.32), with large variations across countries (see Table 3 and Table 4). For example, doctors in Togo and nurses in Uganda have absence rates of 50% (95% CI 0.26-0.74) and 47% (95% CI 0.43-0.50), respectively, while the average absence rate of doctors in Tanzania is 16% (95% CI 0.12-0.21).

# **_Availability of drugs and medical equipment_**

The availability of drugs for treatment of diarrhea (oral rehydration salts, or ORS), postpartum hemorrhage (oxytocin), and pneumonia (amoxycillin or cotrimoxazole) was collected for 9 of the 10 countries and drugs for malaria (artemisinin-based combination 12
therapy, or ACT) for 8 countries. Data on a minimum set of medical equipment (thermometer, stethoscope, sphygmomanometer) were collected for 9 countries. On average, 42% (95% CI 0.40-0.45) of the facilities were stocked with all four (or all three for Kenya) types of drugs and 70% (95% CI 0.69-0.73) of the facilities had a minimum set of functioning medical equipment (see Table S13). ORS is available in 84% (95% CI 0.82-0.87) of the facilities, ACT in 77% (95% CI 0.75-0.80) of the facilities, antibiotics for pneumonia in 69% (95% CI 0.66-0.72) of the facilities and oxytocin in 62% (95% CI 0.60-0.65) of the facilities (see Table 3). There is substantial variation in drug availability across countries (see Table S13 and Figure S14).

# **_Overall care readiness_**

Table 5 reports the probability of care readiness that meets minimum quality standards for five individual conditions and three sets of conditions. To assess how binding to care readiness each of the underlying factors that define the minimum quality standards we use here is, we report three estimates for each condition. Column (1) reports the probability of care readiness which meets minimum quality standards in terms of availability of essential drugs and the probability that providers are present in the facility and knowledgeable to diagnose and treat the condition as measured by the survey. Column (2) of Table 5 reports the probability of care readiness if we assume all facilities have available the essential drugs for the condition(s) in question, but we observe provider availability as it is in the facilities. Column (3) of Table 5 reports the probability of care readiness if we use the availability of drugs as it is but assume that all providers are present.

Table 3 reports summary statistics on the three components used to derive the probability of care readiness, namely (i) whether there is at least one provider working in the clinic that can correctly diagnose and prescribe treatment for the condition(s); (ii) health 13
worker attendance; and (iii) whether or not the facility has a minimum set of drugs to treat the condition(s). The raw data, by country, on vignette performance, drug availability, and provider presence are reported in Tables 1, 2, 4, and S13.

Five results from Table 5 stand out. First, on average, only 14% (95% CI 0.12-0.15) of facilities are ready to provide selected child, neonatal, and maternity care that meets minimum quality standards, with (i) at least one provider available and able to correctly diagnose and treat diarrhea, pneumonia, post-partum hemorrhage and neonatal asphyxia, and (ii) minimum drugs as required (column (1), Table 5). Second, the share of facilities with care readiness that meets minimum quality standards increases to 19% (95% CI 0.18-0.21) when all facilities are assumed to have the essential drugs in stock (column (2), Table 5). That is, given the (low) level of clinical knowledge, even if all facilities were fully stocked with key essential drugs, the share of facilities ready to provide care that meets minimum quality standards would increase by 5 percentage points. Third, addressing absenteeism; i.e., assuming all staff are present during working hours, increases the share of facilities ready to provide care that meets minimum quality standards only by a small margin (less than 3 percentage points) on average (column (3), Table 5). Ultimately, without provider knowledge to diagnose and treat patients, improving either the supply of medicines or the attendance of health providers will accomplish little. Fourth, there is substantial variation in the probability of care readiness that meets minimum quality standards across conditions, from 62% (95% CI 0.60-0.65) for malaria to 34% (95% CI 0.31-0.36) for post-partum hemorrhage (Column (1), Table 5). Fifth, the average outcomes reported in Table 5 mask large variations both within and across countries (see Table S15). In Figure 3 (and Figure S16) we plot the relationships between the probability of care readiness under the assumption of full availability of equipment and drugs versus the probability of care readiness under the conditions we actually observe, and then between the probability of care readiness under the assumption of little absenteeism versus 14
actually observed conditions in order to investigate some of this heterogeneity. For Sierra Leone and Tanzania, the share of facilities meeting minimum quality standards for child, neonatal, and maternity care would increase from 26% to 41% and from 25% to 39%, respectively, if all facilities were stocked with essential drugs. On the other hand, no health workers absenteeism would have a smaller impact in both countries (increasing care readiness from 26% to 31% in Sierra Leone and from 25% to 27% in Tanzania). In Kenya, the share of facilities with care readiness meeting minimum quality standards is estimated to increase from 45% to 60% with no worker absenteeism, while ensuring 100% drugs availability will only increase care readiness in Kenya from 45% to 51%.

In much of health policy, there is an underlying assumption that staffing facilities with “qualified” medical providers (and a basic set of medical equipment) is associated with quality care, hence the use of indicators such as number of doctors per 10,000 population. Figure 4 reports the share of providers accurately diagnosing and treating the two main child killer conditions (diarrhea and pneumonia) by different cadres in all 10 countries. While doctors are more knowledgeable than nurses, a high share of doctors do not possess sufficient knowledge to diagnose and treat the main child killer conditions. On average, 36% (95% CI 0.26-0.46) of doctors and 15% (95% CI 0.11-0.20) of nurses can accurately diagnose and treat both conditions, while 22% (95% CI 0.12-0.31) of the doctors and 34% (95% CI 0.27-0.41) of the nurses fail to accurately diagnose and treat any of the conditions.

Figure S17 illustrates how providers’ ability to diagnose and treat the two main child killer conditions relates to inputs (medical equipment (S17A) and drugs (S17 B and C)). 34% (95% CI 0.32-0.37) of the facilities with functioning equipment and 29% (95% CI 0.24-0.35) of the facilities without such equipment, have at least one provider employed who has the knowledge to diagnose and treat both conditions. Thus, in the majority of facilities, 15
independent of access to functioning equipment, there is no provider who can diagnose and treat both conditions.

50% (95% CI 0.44-0.57) of the facilities with antibiotics in stock have at least one provider employed which has the knowledge to diagnose and treat pneumonia, while in 51% (95% CI 0.44-0.58) of the facilities with ORS in stock, there is at least one provider that has the knowledge to diagnose and treat diarrhea. About half of the facilities have at least one provider employed which has the knowledge to diagnose and treat pneumonia, and about one in four of the facilities have at least one provider employed which has the knowledge to diagnose and treat diarrhea, but they do not have the drugs in stock to treat them (antibiotics and ORS, respectively).

# **Discussion**

This analysis demonstrates the value of measuring clinical knowledge, provider availability, and other structural dimensions of quality in order to provide a diagnostic measure of key challenges, most clearly (and perhaps most damagingly) manifested at the facility level, in a health system. The evidence, and more generally the Service Delivery Indicators (SDI) instrument, provides a useful way for governments to benchmark health systems performance on such measures. The SDI is complementary to other survey instruments, such as the Service Availability and Readiness Assessment (SARA) and the Service Provision Assessment (SPA). The distinguishing characteristics of SDI are the clinical vignettes to assess providers’ clinical knowledge and the use of unannounced visits to quantify absenteeism.

This study uses the lowest common denominator across the country-specific protocols to define comparable care readiness criteria across countries. Moreover, there is the wellknown gap between what health workers can do and what they actually do (the “can-do” gap).<sup>26</sup> For these reasons, we expect that in most surveyed countries the probability of providing minimum quality care according to country-specific guidelines is even lower than 16
that reported in this study. Other studies in the region suggest that the same low performance we observe in vignettes also appear – often to a greater extent – in tests of what providers actually do, with anonymous standardized patients. In Kenya and South Africa, anonymous standardized patients presented with symptoms of tuberculosis: few received an appropriate diagnostic test and most received medication that was either unnecessary or inappropriate.<sup>27</sup> In India, 21 percent of health workers offered potentially harmful diarrhea treatments in vignettes and 72 percent offered them to standardized patients.<sup>28</sup> Thus, even when diagnostic accuracy is high (as for malaria and tuberculosis in our study), treatment may be inaccurate.<sup>27</sup> While we focus on country averages in this study, care readiness may vary dramatically both across regions and within countries. For example, other research on quality of care shows that health workers in South Africa were much less likely to prescribe unnecessary medications for tuberculosis than in India, and in India, correct diagnosis and treatment rates were higher in urban than in rural areas.<sup>27,29</sup> Ultimately, policy action by governments should hinge on appropriate national and sub-national analysis.

Not all misdiagnoses and mistreatments impose equal costs on patients and on health care systems. Some mistreatments – e.g., prescription of antibiotics for viral diarrhea – impose longer term costs on the patient population with greater resistance to antibiotics but limited short-term adverse impacts to the patient. In this analysis, we focus on the countryspecific protocols, but one could imagine an analysis that weighs the direct and indirect cost to patients of different errors in diagnosis and treatment.

Furthermore, there are times that deviations from diagnostic and treatment protocols may be guided by information rather than ignorance. In the case of prescribing antibiotics for diarrhea, recent evidence suggests that antibiotics promote growth among young children, so medical providers could be incorporating that information into their treatment.<sup>30</sup> That said, while a medical provider may overprescribe antibiotics for that reason in practice, there is less 17
reason to expect that she would do so in a vignette as a response to a specific diarrheal condition. In contexts where providers seem to have limited skills, ensuring that guidelines adhere to the best and latest knowledge and practice, and encouraging providers to follow those guidelines, are likely to enhance the quality of care.

The complementarity between different aspects of care is an important dimension to consider when improving the quality of care. For example, despite the fact that the study finds significant levels of provider absenteeism and gaps in medications and equipment, this study suggests that not all of these dimensions merit the same priority. Without provider knowledge of symptoms and treatments, no amount of effort or medication will save a patient’s life. Once health providers have sufficient knowledge and norms or incentives are in place for them to be present in the facility, equipment and medicines become more essential. Ultimately, a health care system that provides quality care will have to meet all of these needs – knowledge, effort, equipment, and essential medicines.

A critical question arising from this work is what actions a health system can take to improve on the effectiveness of its workforce. This research demonstrates that health worker knowledge is a major constraint. Health systems must upgrade the skills of their current and future workers, from doctors to community health workers. The Lancet Commission recommended adopting a competency-based clinical education based on “active learning, early clinical exposure, and problem-solving learning.” The evidence on the costeffectiveness of different modalities through which this can be done remains weak. For example, a major training program – implemented in 76 low- and middle-income countries – had limited impact on appropriate treatment for basic child conditions.<sup>31</sup> The combination of training with supervision or training with group problem solving delivered more promising results.<sup>32</sup> With the increasing penetration of digital technologies, more systems are experimenting with innovations such as mobile devices for training, supervision and 18
mentoring, easy access to decision tools to improve diagnosis and treatment, and the collection of more timely and relevant data. But the use of these technologies in LMICs is still in its infancy and evidence on its effectiveness and costs remains limited.<sup>33</sup> Additional interventions to improve quality of care are more of regulatory and policy nature, such as licensing of providers, accreditation of facilities and oversight of training institutions.

After a decade of evidence from SDI surveys, we identify a set of potential developments that could further strengthen its use and impact in improving quality of care, such as faster and cheaper ways to collect data, attention to a broader set of medical conditions and increased effort to understand drivers of performance differences. As countries increasingly track availability of inputs through health management information systems, SDI could progressively focus more on process indicators of quality. This is in line with the recommendations by the recently published findings from the Lancet Global Health Commission on High Quality Health Systems which identified competent care as one of the key dimensions to measure to advance the quality agenda.

# **Contributors**

JS and WW led the team developing the design of the Service Delivery Indicators instrument. WW supervised data collection in the 10 countries. All authors contributed in drafting and revising the manuscript and approved the final version for submission.

# **Declaration of interests**

We declare no competing interests.

**Funding** The World Bank, Service Delivery Indicators Trust Fund (funded in large part by the

Hewlett Foundation).

# **Acknowledgments**

We are grateful to the many researchers, survey experts and enumerators who have worked on the country surveys that made this cross-country paper possible and are especially grateful for the contributions th <mark>e African Economic Research Consortium.</mark> We are grateful to the World Bank, and in particular the Service Delivery Indicators Trust Fund (funded in large part 19
by the Hewlett Foundation) for supporting the research. JS also thanks Handelsbankens forskningsstiftelse for financial support. The authors would like to thank Amina Mendez Acosta and Evelina Linnros for assistance in this analysis and are grateful for helpful comments from Tessa Bold, Ruben Conner, and Amanda Glassman. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.

20
# **References**

- 1 Kruk ME, Gage AD, Arsenault C, _et al._ High-quality health systems in the Sustainable Development Goals era: time for a revolution. _Lancet Global Health_ 2018; **6** :PE1196E1252. doi:10.1016/S2214-109X(18)30386-3

- 2 The National Academies of Sciences, Engineering and Medicine. Crossing the Global Quality Chasm: Improving Health Care Worldwide: Health and Medicine Division. Washington, DC: The National Academies Press 2018.

- 3 Kieny M-P, Evans TG, Scarpetta S, _et al._ Delivering quality health services: a global imperative for universal health coverage (English) | The World Bank. Washington, DC: World Bank Group 2018.

- 4 Kruk ME, Gage AD, Joseph NT, _et al._ Mortality due to low-quality health systems in the universal health coverage era: a systematic analysis of amenable deaths in 137 countries - The Lancet. _Lancet_ 2018;392:2203–12.

- 5 Björkman M, Svensson J. Power to the People: Evidence from a Randomized Field Experiment on Community-Based Monitoring in Uganda. _Q J Econ_ 2009;124:735–69. doi:10.1162/qjec.2009.124.2.735

- 6 STROBE Statement: Available checklists. https://www.strobestatement.org/index.php?id=available-checklists (accessed 3 Sep 2020).

- 7 Miller GE. Invited Interviews: The Assessment of Clinical Skills/Competence/Performance. _Academic Medicine_ 1990;65:S63–7.

- 8 Kwan A, Daniels B, Saria V, _et al._ Variations in the quality of tuberculosis care in urban India: A cross-sectional, standardized patient study in two cities. _PLOS Medicine_ 2018;15:e1002653. doi:10.1371/journal.pmed.1002653

- 9 King JJC, Das J, Kwan A, _et al._ How to do (or not to do) ... using the standardized patient method to measure clinical quality of care in LMIC health facilities. _Health Policy Plan_ 2019;34:625–34. doi:10.1093/heapol/czz078

- 10  Kwan A, Daniels B, Bergkvist S, _et al_ . Use of standardised patients for healthcare quality research in low- and middle-income countries. _BMJ Global Health_ 2019;4:e001669. doi:10.1136/bmjgh-2019-001669

- 11  Bold T, Gauthier B, Svensson J, _et al._ Delivering service indicators in education and health in Africa: a proposal. World Bank 2010.

- 12  Bold T, Svensson J, Gauthier B, _et al._ Service delivery indicators: Pilot in education and health care in Africa. Bergen, Norway: Chr. Michelsen Institute 2011.

- 13 Atakhodjaeva R, Rockmore C. Service Delivery Indicators Report - Niger. World Bank 2017. http://documents.worldbank.org/curated/en/833401504769240083/Niger-Servicedelivery-indicators-health (accessed 2 Mar 2020).

- 14  Martin GH, Pimhidzai O. Service Delivery Indicators Report - Kenya. World Bank 2013. https://www.sdindicators.org/countries/kenya (accessed 2 Mar 2020).

21
- 15 Wane W, Martin GH. Service Delivery Indicators Report - Uganda. World Bank 2013. https://www.sdindicators.org/countries/uganda (accessed 2 Mar 2020).

- 16  World Bank. Service Delivery Indicators Report - Senegal. World Bank 2012. https://www.sdindicators.org/countries/senegal (accessed 2 Mar 2020).

- 17 World Bank. Service Delivery Indicators Report - Tanzania. World Bank 2016. http://documents.worldbank.org/curated/en/162561467996712439/Tanzania-Health (accessed 2 Mar 2020).

- 18  Martin G, Shireen M. Kenya Public Expenditure Tracking and Service Delivery Indicator Survey (PETS-plus), Health Questionnaire 2012. World Bank 2017. https://microdata.worldbank.org/index.php/catalog/2751/related-materials

- 19  Pimhidzai O. Nigeria Service Delivery Indicator Health 2013. Worldbank 2017. https://microdata.worldbank.org/index.php/catalog/2559/related-materials

- 20  Wane W. Tanzania Service Delivery Indicators (SDI) Survey Health Questionnaires 2014. World Bank 2017.

- https://microdata.worldbank.org/index.php/catalog/2582/related-materials

- 21  Rockmore C. Togo Enquete sur les Indicateurs de Prestations de Services (SDI), Santé 2013. World Bank 2017. https://microdata.worldbank.org/index.php/catalog/2560/related-materials

- 22  Wane W. Uganda Service Delivery Indicator Health Questionnaire, Module H1 - Service Delivery 2013. World Bank 2017. https://microdata.worldbank.org/index.php/catalog/2750/related-materials

- 23  Peabody JW, Luck J, Glassman P, _et al._ Comparison of Vignettes, Standardized Patients, and Chart Abstraction: A Prospective Validation Study of 3 Methods for Measuring Quality. _JAMA_ 2000;283:1715–22. doi:10.1001/jama.283.13.1715

- 24  Peabody JW, Luck J, Glassman P, _et al._ Measuring the Quality of Physician Practice by Using Clinical Vignettes: A Prospective Validation Study. _Ann Intern Med_ 2004;141:771. doi:10.7326/0003-4819-141-10-200411160-00008

- 25  Chaudhury N, Hammer J, Kremer M, _et al._ Missing in Action: Teacher and Health Worker Absence in Developing Countries. _Journal of Economic Perspectives_ 2006;20:91–116. doi:10.1257/089533006776526058

- 26  Ibnat F, Leonard KL, Bawo L, _et al._ The Three-Gap Model of Health Worker Performance. The World Bank 2019.

- 27  Daniels B, Kwan A, Pai M, _et al_ . Lessons on the quality of tuberculosis diagnosis from standardized patients in China, India, Kenya, and South Africa. _Journal of Clinical Tuberculosis and Other Mycobacterial Diseases_ 2019;16:100109. doi:10.1016/j.jctube.2019.100109

- 28 Mohanan M, Vera-Hernández M, Das V, _et al._ The know-do gap in quality of health care for childhood diarrhea and pneumonia in rural India. JAMA pediatrics Published Online First: 2015. doi:10.1001/jamapediatrics.2014.3445

22
- 29  Das J, Holla A, Das V, et al. In Urban And Rural India, A Standardized Patient Study Showed Low Levels Of Provider Training And Huge Quality Gaps. Health Affairs 2012;31:2774–84. doi:10.1377/hlthaff.2011.1356

- 30  Gough EK, Moodie EEM, Prendergast AJ, _et al._ The impact of antibiotics on growth in children in low and middle income countries: systematic review and meta-analysis of randomised controlled trials. _BMJ_ 2014;348. doi:10.1136/bmj.g2267

- 31  Rowe AK, Rowe SY, Holloway KA, _et al._ Does shortening the training on Integrated Management of Childhood Illness guidelines reduce its effectiveness? A systematic review. _Health Policy Plan_ 2012;27:179–93. doi:10.1093/heapol/czr033

- 32  Rowe AK, Labadie G, Jackson D, _et al._ Improving health worker performance: an ongoing challenge for meeting the sustainable development goals. _BMJ_ 2018;362. doi:10.1136/bmj.k2813

- 33  Free C, Phillips G, Galli L, _et al._ The Effectiveness of Mobile-Health Technology-Based Health Behaviour Change or Disease Management Interventions for Health Care Consumers: A Systematic Review. _PLOS Medicine_ 2013;10:e1001362. doi:10.1371/journal.pmed.1001362

23
**Table 1.** Diagnostic accuracy by country

||(1)<br>Diarrhea|(2)<br>Pneumonia|(3)<br>Malaria|(4)<br>Post-partum<br>hemorrhage|(5)<br>Neonatal<br>asphyxia|(6)<br>Tuberculosis|(7)<br>Diabetes|(8)<br>All|
|---|---|---|---|---|---|---|---|---|
|Kenya|0.76<br>(.74-.77)<br>[4505]|0.78<br>(.77-.80)<br>[4491]|NA|0.88<br>(.87-.89)<br>[4481]|0.86<br>(.85-.87)<br>[4476]|0.96<br>(.95-.96)<br>[4488]|0.58<br>(.56-.59)<br>[4489]|0.80<br>(.80-81)<br>[26,930]|
|Madagascar|0.18|0.38|0.96|0.55|0.52|0.84|0.12|0.51|
||(.12-23)|(.31-.45)|(.94-.97)|(.47-.62)|(.45-.60)|(.78-.90)|(.08-.16)|(.48-53)|
||<br>[642]|<br>[642]|<br>[642]|<br>[642]|<br>[642]|<br>[642]|<br>[642]|<br>[4,494]|
|Mozambique|0.73|0.66|0.96|0.76|0.60|0.86|0.34|0.70|
||(.69-76)|(.62-.70)|(.95-.98)|(.72-79)|(.56-.64)|(.83-.89)|(.30-.38)|(.69-.72)|
||<br>[725]|<br>[725]|<br>[725]|<br>[724]|<br>[724]|<br>[725]|<br>[725]|<br>[5,073]|
|Niger|0.92|0.47|0.97|0.62|0.58|0.74|0.11|0.63|
||(.90-95)|(.41-.54)|(.95-.98)|(.65-.68)|(.52-.65)|(.68-.80)|(.07-.14)|(.61-.65)|
||<br>[519]|<br>[519]|<br>[519]|<br>[601]|<br>[601]|<br>[519]|<br>[519]|<br>[3,797]|
|Nigeria|0.26<br>(.24-27)|0.42<br>(.41-.44)|0.88<br>(.87-.89)|0.52<br>(.51-.54)|0.36<br>(.34-.37)|0.58<br>(.56-.60)|0.30<br>(.28-.32)|0.47<br>(.47-.48)|
||<br>[4711]|<br>[4719]|<br>[4669]|<br>[4505]|<br>[4386]|<br>[4676]|<br>[4628]|<br>[32,294]|
|Senegal|0.63<br>(.49-.77)|0.56<br>(.39-.73)|0.04<br>(.00-.08)|NA|NA|0.73<br>(.61-.85)|NA|0.49<br>(.41-.57)|
||<br>[152]|<br>[152]|<br>[152]|||<br>[152]||<br>[608]|
|Sierra Leone|0.95<br>(.93-.97)|0.57<br>(.53-.61)|0.99<br>(.99-1.0)|0.94<br>(.92-.95)|0.86<br>(.84-.90)|0.88<br>(.86-.91)|0.25<br>(.22-.28)|0.78<br>(.77-.79)|
||<br>[826]|<br>[824]|<br>[824]|<br>[824]|<br>[824]|<br>[824]|<br>[824]|<br>[5,770]|
|Tanzania|0.87|0.75|0.97|0.89|0.81|0.88|0.46|0.80|
||(.82-.93)|(.69-.80)|(.95-.99)|(.84-.94)|(.76-.86)|(.84-.92)|(.39-.52)|(.78-.82)|
||<br>[542]|<br>[542]|<br>[542]|<br>[542]|<br>[542]|<br>[542]|<br>[542]|<br>[3,794]|
|Togo|0.18|0.61|0.97|0.58|0.11|0.85|0.53|0.55|
||(.09-.26)|(.49-.73)|(.95-.99)|(.45-.70)|(.04-.18)|(.76-.93)|(.41-.66)|(.50-.60)|
||<br>[302]|<br>[302]|<br>[302]|<br>[298]|<br>[298]|<br>[302]|<br>[302]|<br>[2,106]|
|Uganda|<br>0.39|<br>0.47|<br>0.94|<br>0.65|<br>0.58|<br>0.78|<br>0.49|<br>0.61|
||(34-44)|(43-52)|(92-96)|(60-69)|(53-63)|(74-82)|(44-53)|(60-63)|
||..<br>[709]|..<br>[709]|..<br>[708]|..<br>[709]|..<br>[707]|..<br>[709]|..<br>[709]|..<br>[4,960]|
|All|<br>0.59|<br>0.57|<br>0.85|<br>0.71|<br>0.59|<br>0.81|<br>0.35|<br>0.64|
||(.56-.61)|(.54-.59)|(.82-.89)|(.69-.73)|(.57-.61)|(.79-.83)|(.33-.37)|(.62-.65)|
||<br>[13,633]|<br>[13,625]|<br>[9,083]|<br>[13,326]|<br>[13,198]|<br>[13,579]|<br>[13,380]|<br>[89,824]|

Notes: Mean diagnostic accuracy: i.e., the percentage of health workers who correctly diagnose the condition, by country and condition, with all individual country means calculated using country-specific sampling weights, and the (unweighted) mean across conditions for each country in column 8. 95% confidence intervals in parenthesis and 24
number of observations in brackets. For Nigeria and Togo, the answer reflects whether providers were able to provide a correct diagnosis for diarrhea and dehydration presented jointly.

25
**Table 2.** Diagnostic and treatment accuracy by country

||(1)<br>Diarrhea|(2)<br>Pneumonia|(3)<br>Malaria|(4)<br>Post-partum<br>hemorrhage|(5)<br>Neonatal<br>asphyxia|(6)<br>Tuberculosis|(7)<br>Diabetes|(8)<br>All|
|---|---|---|---|---|---|---|---|---|
|Kenya|0.69<br>(.67-70)<br>[4505]|0.65<br>(.63-.66)<br>[4491]|NA|0.71<br>(.69-.72)<br>[4481]|0.79<br>(.78-81)<br>[4476]|0.70<br>(.69-.71)<br>[4488]|0.49<br>(.47-.50)<br>[4489]|0.67<br>(.66-.68)<br>[26,930]|
|Madagascar|0.17|0.28|0.86|0.22|0.50|0.19|0.11|0.33|
||(.11-22)|(.22-.34)|(.82-.89)|(.16-29)|(.43-.57)|(.13-.25)|(.07-.15)|(.30-.36)|
||<br>[642]|<br>[642]|<br>[642]|<br>[642]|<br>[642]|<br>[642]|<br>[642]|<br>[4,494]|
|Mozambique|0.68|0.45|0.83|0.30|0.57|0.51|0.23|0.51|
||(.64-.72)|(.41-.49)|(.79-.86)|(.26-.34)|(.53-.62)|(.47-.55)|(.20-.27)|(.49-.53)|
||<br>[725]|<br>[725]|<br>[725]|<br>[724]|<br>[724]|<br>[725]|<br>[725]|<br>[5,073]|
|Niger|0.84<br>(.79-.89)|0.42<br>(.35-.48)|0.78<br>(.73-.83)|0.19<br>(.14-.24)|0.48<br>(.42-.54)|0.21<br>(.16-.26)|0.07<br>(.05-.10)|0.42<br>(.40-.45)|
||<br>[519]|<br>[519]|<br>[519]|<br>[601]|<br>[601]|<br>[519]|<br>[519]|<br>[3,797]|
|Nigeria|0.23<br>(.22-.25)|0.29<br>(.27-30)|0.59<br>(.57-.61)|0.21<br>(.20-.23)|0.32<br>(.30-.33)|0.16<br>(.15-.18)|0.19<br>(.18-.21)|0.28<br>(.28-.29)|
||<br>[4693]|<br>[4572]|<br>[4669]|<br>[4387]|<br>[4383]|<br>[4669]|<br>[4609]|<br>[31,982]|
|Senegal|0.55<br>(.39-.70)|0.43<br>(.28-.58)|0.04<br>(.00-.08)|NA|NA|0.21<br>(.12-.30)|NA|0.31<br>(.24-.38)|
||<br>[152]|<br>[152]|<br>[152]|||<br>[152]||<br>[608]|
|Sierra Leone|0.89|0.45|0.92|0.67|0.85|0.33|0.21|0.62|
||(.87-.92)|(.42-.49)|(.89-.94)|(63-.70)|(.82-.88)|(.29-.36)|(.18-.24)|(.60-.63)|
||<br>[826]|<br>[824]|<br>[824]|<br>[824]|<br>[824]|<br>[824]|<br>[824]|<br>[5,770]|
|Tanzania|0.85|0.57|0.9|0.36|0.79|0.41|0.41|0.61|
||(.80-.90)|(.50-.63)|(.86-.93)|(.30-.42)|(.74-.84)|(.35-.47)|(.34-.47)|(.59-.64)|
||<br>[542]|<br>[542]|<br>[542]|<br>[542]|<br>[542]|<br>[542]|<br>[542]|<br>[3,794]|
|Togo|<br>0.17|<br>0.57|<br>0.74|<br>0.09|<br>0.10|<br>0.11|<br>0.52|<br>0.33|
||(.08-.25)|(.45-.69)|(.64-.85)|(.02-.15)|(.04-.17)|(.07-.16)|(.39-.64)|(.28-.38)|
||<br>[302]|<br>[302]|<br>[302]|<br>[298]|<br>[298]|<br>[302]|<br>[302]|<br>[2,106]|
|Uganda|<br>0.29|<br>0.25|<br>0.61|<br>0.27|<br>0.50|<br>0.27|<br>0.39|<br>0.37|
||(24-33)|(21-29)|(56-65)|(23-31)|(45-55)|(23-31)|(35-44)|(35-39)|
||..<br>[709]|..<br>[709]|..<br>[708]|..<br>[709]|..<br>[707]|..<br>[709]|..<br>[709]|..<br>[4,960]|
|All|<br>0.54|<br>0.43|<br>0.70|<br>0.34|<br>0.55|<br>0.31|<br>0.29|<br>0.45|
||(.51-.56)|(.41-.46)|(.66-.73)|(.32-.35)|(.52-.58)|(.29-.33)|(.27-.31)|(.43-.46)|
||<br>[13,518]|<br>[13,381]|<br>[8,988]|<br>[13,120]|<br>[13,110]|<br>[13,477]|<br>[13,267]|<br>[88,862]|

Notes: Mean diagnostic accuracy: i.e., the percentage of health workers who correctly diagnose the condition, by country and condition, with all individual country means calculated using country-specific sampling weights, and the (unweighted) mean across conditions for each country in column 8. 95% confidence intervals in parenthesis and 26
number of observations in brackets. For Nigeria and Togo, the answer reflects whether providers were able to provide a correct diagnosis/treatment for diarrhea and dehydration presented jointly.

27
**Table 3:** Care readiness: Knowing how to diagnose and treat, health worker attendance, and availability of drugs

||(1)<br>All|(2)<br>95% CI|(3)<br>No. of<br>facilities.|
|---|---|---|---|
|_Share of clinics with at least one healt_<br>_prescribe treatment for:_|_h worker that ca_|_n correctly diag_|_nose and_|
|Diarrhea|68.7%|(.66-.72)|7,910|
|Pneumonia|61.5%|(.59-.64)|7,910|
|Malaria|90.2%|(.88-.93)|4,799|
|Neonatal asphyxia|70.5%|(.68-.73)|7,863|
|Post-partum hemorrhage|51.3%|(.49-.54)|7,863|
|Child care|43.4%|(.41-.46)|7,863|
|Neonatal & maternal care|44.1%|(.42-.47)|7,863|
|Child, neonatal & maternal care|21.7%|(.20-.23)|7,863|
|_Health worker attendance:_||||
|Doctors|71.0%|(.64-.78)|1,223|
|Clinical officers|65.0%|(.59-.71)|2,277|
|Nurses|70.9%|(.68-.73)|5,381|
|Community health workers|74.0%|(.71-.77)|3,778|
|_Drugs available:_||||
|Oral rehydration salts (ORS)|84.4%|(.82-.87)|7,856|
|Antibiotics for pneumonia|69.1%|(.66-.72)|7,854|
|ACT|77.5%|(.75-.80)|4,790|
|Oxytocin|62.3%|(.60-.65)|6,438|
|ORS, antibiotics, oxytocin|48.6%|(.46-.51)|6,436|

**Notes** : Column (1) reports the underlying components of the care readiness estimation reported in Table 5. The unit of analysis is the facility. The estimates in the tables are (unweighted) mean outcomes across countries, with the country means calculated using country-specific sampling weights. Column (2) reports number of facilities. See notes to Table 5 for details.

28
**Table 4.** Absence rate by country

||(1)|(2)|(3)|
|---|---|---|---|
||Absence rate<br>(all)|Absence rate<br>(doctors)|Absence rate<br>(nurses)|
|Kenya|0.46<br>|0.41<br>|0.47<br>|
||(.45-.46)|(.40-.43)|(.46-.48)|
||[7838]|[2139]|[5699]|
|Madagascar|0.24|0.28|0.22|
||(.20-.28)|(.21-.35)|(.18-.26)|
||[1340]|[574]|[739]|
|Mozambique|0.22|0.22|0.22|
||(.20-.25)|(.18-.27)|(.19-.25)|
||[800]|[262]|[410]|
|Niger|0.27|0.38|0.30|
||(.22-.32)|(.20-.55)|(.25-.36)|
||[547]|[62]|[354]|
|Nigeria|0.28<br>|0.24<br>|0.29<br>|
||(.27-.29)|(.17-.30)|(.27-.31)|
||[6724]|[426]|[1788]|
|Sierra Leone|0.27|0.28|0.30|
||(.25-.28)|(.22-.35)|(.27-.33)|
||[1663]|[204]|[554]|
|Tanzania|0.14|0.16|0.15|
||(.12-.16)|(.12-.21)|(.12-.19)|
||[1883]|[513]|[776]|
|Togo|0.40|0.50|0.36|
||(.31-.48)|(.26-.74)|(.26-.46)|
||[487]|[82]|[257]|
|Uganda|0.45|0.47|0.47|
||(.42-.48)|(.40-.55)|(.43-.50)|
||<br>[1218]|<br>[257]|<br>[667]|
|All|0.30|0.33|0.31|
||(.29-.32)|(.29-.37)|(.29-.32)|
||[22,341]|[4,481]|11,163|

Note: The table reports the mean absence rate for all staff with some medical training by country. Column (1): All staff with medical training (doctors, clinical officers, nurses, and community health workers); column (2): doctors and clinical officers; column (3): nurses. All individual country statistics are calculated using country-specific sampling weights. 95% confidence intervals in parenthesis and number of observations in parenthesis. A provider is marked as absent from a facility if, during the second unannounced visit, the provider is not found anywhere on the facility premises. Otherwise, the provider is marked as present.

29
**Table 5** : Probability of care that meets minimum quality standards (%)

||(1)<br>|(2)<br>Probability of<br>|(3)<br>Care Readiness<br>|(4)<br>|
|---|---|---|---|---|
|_Condition_|As observed|Assume<br>available<br>drugs|Assume<br>provider<br>presence|No. of<br>facilities|
|Diarrhea|51.9<br>(49.2-54.5)|57.7<br>(54.9-60.4)|61.7<br>(58.9-64.5)|7,856|
|Pneumonia|39.3<br>(36.9-41.7)|50.9<br>(48.4-53.5)|46.7<br>(44.1-49.3)|7,854|
|Malaria|62.5<br>(59.6-65.4)|77.9<br>(75.3-80.6)|71.6<br>(68.7-74.5)|4,790|
|Neonatal asphyxia|58.8<br>(56.2-61.5)|-|70.5<br>(67.6-73.4)|7,863|
|Post-partum hemorrhage|33.7<br>(31.5-35.8)|43.2<br>(40.8-45.7)|39.5<br>(37.1-41.8)|6,438|
|Child care|27.9<br>(25.9-29.9)|36.4<br>(34.1-38.6)|33.1<br>(30.9-35.4)|7,854|
|Neonatal & maternal care|29.0<br>(27.0-30.9)|37.1<br>(34.9-39.4)|34.2<br>(32.0-36.4)|6,438|
|Child, neonatal & maternal care|13.7<br>(12.5-14.9)|19.1<br>(17.6-20.6)|16.6<br>(15.2-17.9)|6,436|

**Notes** : Column (1) reports the estimated probability of care readiness that meets minimum quality standards for different conditions (or set of conditions). Column 2 reports the estimated probability of care readiness that meets minimum quality standards, assuming essential drug(s) (oral rehydration salts or rehydration therapy for diarrhea; antibiotics for pneumonia; artemisinin-based combination therapy (ACT) for malaria; and oxytocin for postpartum hemorrhage) for treating the condition(s) are available. No essential drugs data were collected for neonatal asphyxia. Column 3 reports the estimated probability of care readiness that meets minimum quality standards, assuming no absenteeism. Child care includes two conditions (diarrhea and pneumonia). Neonatal & maternity care includes two conditions (post-partum hemorrhage and neonatal asphyxia). Child, neonatal & maternity care includes four conditions (diarrhea, pneumonia, post-partum hemorrhage, and neonatal asphyxia). The estimates in the tables are (unweighted) mean outcomes across countries, with the country means calculated using countryspecific sampling weights, with 95% confidence intervals in parenthesis (see text for details). Data are from clinical vignettes, unannounced visits, and visual inspections from Kenya (2018), Madagascar (2016), Mozambique (2014), Nigeria (2013), Niger (2017), Sierra Leone (2018), Tanzania (2016), Togo (2014), and Uganda (2013), with number of facilities reported in column 4. The malaria vignette was not used in the Kenya (2018) survey.

30
Panel A: Percent of providers diagnosing correctly <!-- Start of picture text --> Diabetes<br>Pneumonia<br>Diarrhea<br>Neonatal asphyxia<br>Post-partum hemorrhage<br>Tuberculosis<br>Malaria<br>0 20 40 60 80 100<br>Providers diagnosing correctly (%)<br><!-- End of picture text --> Panel B: Percent of providers diagnosing and treating correctly <!-- Start of picture text --> Diabetes<br>Tuberculosis<br>Post-partum hemorrhage<br>Pneumonia<br>Diarrhea<br>Neonatal asphyxia<br>Malaria<br>0 20 40 60 80 100<br>Providers diagnosing and treating correctly (%)<br><!-- End of picture text -->

**Figure 1: Diagnostic and treatment accuracy**

Dots represent country-specific means, calculated using country-specific sampling weights, vertical bars indicate mean performance across countries, and boxes delineate the interquartile range. Panel A: Percent of providers diagnosing correctly. Panel B: Percent of providers diagnosing and treating correctly. Data are from clinical vignettes from Kenya (2018), Madagascar (2016), Mozambique (2014), Nigeria (2013), Niger (2017), Senegal (2010), Sierra Leone (2018) Tanzania (2016), Togo (2014), and Uganda (2013). The malaria vignette was not used in the Kenya (2018) survey and the post-partum hemorrhage, neonatal asphyxia, and diabetes vignettes were not used in the Senegal (2010) survey.

31
<!-- Start of picture text --> Diarrhoea correctly diagnosed<br>Diarrhoea incorrectly diagnosed<br>0 10 20 30 40 50 60<br>Inappropriate prescription of antibiotics: diarrhoea (%)<br><!-- End of picture text -->

**Figure 2: Overprescription of antibiotics**

Dots represent country-specific means, calculated using country-specific sampling weights, vertical bars indicate mean performance across countries, and boxes delineate the interquartile range. Data are from clinical vignettes from Kenya (2018), Madagascar (2016), Mozambique (2014), Nigeria (2013), Niger (2017), Senegal (2010), Sierra Leone (2018) Tanzania (2016), Togo (2014), and Uganda (2013). See Appendix Table S11 for the values in this figure.

32
Panel A: Relationship between probability of care readiness under the assumption of full availability of equipment and drugs versus actual conditions observed at the facility, country by country <!-- Start of picture text --> KEN<br>50<br>40 SLE<br>TZA<br>30<br>20 MOZ<br>10<br>NER<br>NGA<br>UGA<br>MDG<br>0 TGO<br>0 10 20 30 40 50<br>Pr. of min. quality standards (%)<br>versus actual conditions observed at the facility, country by country<br>60 KEN<br>50<br>40<br>SLE<br>30<br>TZA<br>20<br>MOZ<br>10<br>NER<br>NGA<br>UGAMDG<br>0 TGO<br>0 10 20 30 40 50 60<br>Pr. of min. quality standards (%)<br>Pr. of min. quality standards: Drugs available (%)<br>Pr. of min. quality standards: No absence (%)<br><!-- End of picture text --> Panel B: Relationship between probability of care readiness under the assumption of low provider absenteeism versus actual conditions observed at the facility, country by country

**Figure 3: Comparing estimates of minimum quality care**

Panel A plots the relationship between the estimated probability of care readiness that meets minimum quality standards for child, neonatal, and maternity care, vs. the estimated probability of care readiness that meets minimum quality standards for child, neonatal, and maternity care assuming essential drugs treating the conditions are available. Panel B plots the relationship between the estimated probability of care readiness that meets minimum quality standards for child, neonatal, and maternity care, vs. the estimated probability of care readiness that meets minimum quality standards for child, neonatal, and maternity care assuming no absenteeism. Child, neonatal, and maternity care includes diarrhea, pneumonia, post-partum hemorrhage and neonatal asphyxia. Essential drugs are oral rehydration salts for diarrhea, antibiotics for pneumonia, and oxytocin for postpartum hemorrhage. Data are from clinical vignettes, unannounced visits, and visual inspections from Kenya (2018), Madagascar (2016), Mozambique (2014), Nigeria (2013), Niger (2017), Sierra Leone (2018), Tanzania (2016), Togo (2014), and Uganda (2013). ISO 3-digit alphabetic codes are: KEN (Kenya), MDG (Madagascar), MOZ (Mozambique), NER (Niger), NGA (Nigeria), SLE (Sierra Leone), TZA (Tanzania), TGO (Togo), UGA (Uganda).

33
<!-- Start of picture text --> 60<br>50<br>40<br>30<br>20<br>10<br>0<br>Community health workers Nurses Doctors<br>No condition One condition Both conditions<br>Share of providers<br><!-- End of picture text -->

**Figure 4: Diagnostic and treatment accuracy across cadres**

Share of providers among doctors, nurses, and community health workers correctly diagnosing the two main killer diseases for children (diarrhea, pneumonia). The estimates are (unweighted) mean outcomes across countries, with the country means calculated using country-specific sampling weights. The error bars represent the 95-percent confidence interval. Data are from clinical vignettes from Kenya (2018), Madagascar (2016), Mozambique (2014), Nigeria (2013), Niger (2017), Sierra Leone (2018) Tanzania (2016), Togo (2014), and Uganda (2013).

34
# **Supplementary appendix**

Supplement to the article, “An analysis of clinical knowledge, absenteeism, and available of resources for maternal and child health: A cross-sectional quality of care study in 10 African countries”

## **List of contents**

S1: STROBE Statement for cross-sectional, observational studies, with information for this study Table S2: How the Study Countries Compare with Sub-Saharan Africa as a Whole Table S3. Disability adjusted life years lost to the diseases measured in the Service Delivery Indicators S4. Vignettes: Development, reason for use, validation, and administration Table S5: Initial scenarios for vignettes S6: Cross-country variation in treatment protocols for the vignettes Table S7: Minimum knowledge S8. Selection of staff, drugs, and equipment for surveying S9. Estimation of the probability of care readiness Figure S10. Variation across the quality spectrum Figure S11. Clustering of Diagnostic and Treatment Accuracy Table S12. Overprescription of antibiotics Table S13. Access to drugs and medical equipment by country Figure S14. Drugs and medical equipment at the facility Table S15. Probability of care that meets minimum quality standards (%): by country Figure S16. Comparing estimates of care readiness Figure S17. Share of providers who provide accurate diagnosis and treatment conditional on availability of equipment and drugs 35
# **S1: STROBE Statement for cross-sectional, observational studies, with information for this study**

||**Item**<br>**No**|**Recommendation**|
|---|---|---|
|**Title and abstract**|1|(a) Indicate the study’s design with a commonly used term in the title or<br>the abstract<br>_In the abstract, we include the following text indicating this is cross-_<br>_sectional data analysis: “With nationally representative, cross-sectional_<br>_data from ten countries in Sub-Saharan Africa, collected using clinical_<br>_vignettes, unannounced visits, and visual inspections, we assess whether_<br>_health providers are available and have sufficient knowledge and means to_<br>_diagnose and treat patients suffering from conditions amenable to primary_<br>_health care.”_|
|||(b) Provide in the abstract an informative and balanced summary of what<br>was done and what was found<br>_In the abstract, we include data on methods and on results._|
|**Introduction**<br>Background/rationale|2|Explain the scientific background and rationale for the investigation being<br>reported<br>_We include the following text: “Measuring quality at various levels of the_<br>_health system—especially where the system fails the people in need—_<br>_remains rare in LMICs. Existing indicators of structural quality, such as_<br>_availability of medicine, equipment, and qualified health attendants, are_<br>_not sufficient to adequately capture the quality of care being offered.1 The_<br>_lack of quality measurement can help to explain both why there is limited_<br>_research on how to effectively transform low-quality health systems into_<br>_high-quality ones, and why there are few such successful reforms in LMICs._<br>_Actionable and credible data on quality, if properly disseminated, can play_<br>_an important role in holding country health systems accountable for_<br>_effective and safe care and can form the basis for quality improvement.”_|
|Objectives|3|State specific objectives, including any prespecified hypotheses<br>_We include the following text: “We use the data to measure dimensions we_<br>_would argue are essential for providing quality care: What do providers_<br>_know? Are providers available at health facilities? And what means do they_<br>_have at their disposal?”_|
|**Methods**|||
|Study design|4|Present key elements of study design early in the paper<br>_We lay out the design: “Our paper reports on the first large scale_<br>_assessments of clinical knowledge, health provider absenteeism, and_<br>_means to provide care in Sub-Saharan Africa. We use data collected_<br>_through clinical vignettes, unannounced visits, and visual inspections of_<br>_over 20,000 providers in over 8,000 facilities from ten Sub-Saharan African_|

36
||_countries, which together represent over 40 percent of the region’s total_<br>_population.”_|
|---|---|
|Setting|5<br>Describe the setting, locations, and relevant dates, including periods of<br>recruitment, exposure, follow-up, and data collection<br>_See the relevant text: “We draw upon data from the Service Delivery_<br>_Indicators (SDI) program—an ongoing program with the aim of collecting_<br>_informative and standardized measures of what providers know, what they_<br>_do, and what they have to work with...To date, the SDI program has_<br>_collected data from ten countries: Kenya (2018), Madagascar (2016),_<br>_Mozambique (2014), Nigeria (2013), Niger (2017), Senegal (2010), Sierra_<br>_Leone (2018), Tanzania (2016), Togo (2014), and Uganda (2013), and has_<br>_since expanded to countries outside the Africa region. While every context_<br>_is different, these ten countries serve as a reasonable proxy for much of_<br>_Sub-Saharan Africa, with an average income per capita, life expectancy,_<br>_and infant mortality similar to those of the continent as a whole (Appendix_<br>_Table S2).”_|
|Participants|6<br>(a) Give the eligibility criteria, and the sources and methods of selection of<br>participants<br>_See the relevant text: “All country datasets were representative at the_<br>_national level except for Nigeria which, mainly owing to security concerns,_<br>_covered 12 out of 37 states and was representative at the state level. The_<br>_surveys used a multistage cluster-sampling design to ensure data were_<br>_representative of rural and urban areas, private and public facilities, and of_<br>_each facility type (e.g., front-line clinics versus hospitals).”_|
|Variables|7<br>Clearly define all outcomes, exposures, predictors, potential confounders,<br>and effect modifiers. Give diagnostic criteria, if applicable<br>_See the relevant text: “There are three essential conditions for quality care._<br>_First, providers have sufficient knowledge to diagnose and treat patients_<br>_suffering from conditions that are amenable to care. Second, they are_<br>_available to provide care when patients seek service. Third, providers have_<br>_the means (e.g., equipment and drugs) to diagnose and treat patients.”_<br>_“Combining data from the clinical vignettes (to measure knowledge) with_<br>_data from the unannounced visit (to measure the extent to which providers_<br>_are present in the facility at regular opening hours), and visual inspections_<br>_(to quantify the availability of drugs and equipment), we estimate the_<br>_probability that a facility is ready to provide care that meets minimum_<br>_quality standards. A facility is defined as being ready to provide care that_<br>_meets minimum quality standards for a given condition if there is at least_<br>_one provider present in the clinic at a given day that can correctly diagnose_<br>_and treat that condition, and the facility has a minimum set of drugs to_<br>_treat it.”_|

37
Data sources/ measurement

- 8* For each variable of interest, give sources of data and details of methods of assessment (measurement). Describe comparability of assessment methods if there is more than one group

_See the relevant text: “Diagnostic and treatment accuracy (knowledge) were measured using clinical vignettes focused on seven common conditions associated with the main causes of child and maternal mortality in Sub-Saharan Africa: malaria, tuberculosis, diarrhea, pneumonia, diabetes, neonatal asphyxia and post-partum hemorrhage. All seven of these are major global health challenges, with three of them – diarrhea, neonatal asphyxia, and malaria – together causing more than 180,000,000 disability-adjusted life years (Appendix Table S3). Whether providers are available to provide care at regular opening hours was assessed by verifying their presence through unannounced visits at the health facilities. Availability of drugs and functionality of equipment was assessed by visual inspections of storage facilities and consulting rooms at the health facilities._ _Clinical vignettes have been used extensively in the medical literature to assess clinical knowledge of health workers. Data from vignettes studies are directly policy relevant as they are closely linked to the quality of medical training and quality improvement programs. Validation studies show that clinical vignettes, for relatively simple scenarios such as those assessed as part of the SDI program, perform similarly to standardized patients methods. Although a health worker’s clinical vignette performance is not a measure of quality of care per se, it represents the highest level of quality care which s/he can deliver given his/her current knowledge, assuming that there are no constraints in drugs, equipment and motivations. (More details on the clinical vignettes are provided in Section S4 in the appendix.)_ _The clinical vignettes were adjusted to country-specific diagnostic and treatment protocols. All health providers assessed were asked for informed consent to participate. The enumerators presented the provider with the patient’s symptoms and prompted the provider to take the patient’s history, suggest physical examination and diagnostic tests that in principle could be conducted on site, make a diagnosis, and prescribe a course of treatment and follow-up management plan. For this study, we identified the lowest common denominators across the country-specific protocols to define comparable criteria across countries for diagnostic and treatment accuracy for each of the conditions. This methodological choice implies that knowledge is assessed at a lower standard than more stringent World Health Organization (WHO) protocols. Based on the responses to each vignette, we generated two variables: diagnostic accuracy and treatment accuracy, which are the percentage of respondents who provided an accurate diagnosis and – separately – an accurate treatment for a given condition. Both outcomes should be viewed as measuring the minimum required knowledge to prescribe correct diagnostic and treatment services. (See Table S7 for information on the coding of the two knowledge scores.)_ 38
_These outcomes do not represent the ideal level of knowledge; for example, they do not account for co-morbidities._

# _To quantify whether providers are_ **_available to provide medical service_**

**_when patients seek care_** _, we followed a standard procedure in which enumerators visited each facility twice. During the first visit, information on all staff working at the facility, by professional cadre, was collected. (See Section S5 in the appendix for details on the selection of staff,_ **_drugs, and equipment_** _for surveys.) A few days after the first visit, enumerators made a second unannounced visit to the facility and assessed the presence of a randomly pre-selected list of providers (or all providers, if less than 10 providers were available). Only staff on duty that day were included in the absenteeism measure. Country-level averages for each assessment were calculated using country-specific sampling weights._

_Combining data from the clinical vignettes (to measure knowledge) with data from the unannounced visit (to measure the extent to which providers are present in the facility at regular opening hours), and visual inspections (to quantify the availability of drugs and equipment), we estimate the probability that a facility is ready to provide care that meets minimum quality standards.”_

- Bias 9 Describe any efforts to address potential sources of bias

_In order to minimize bias, the study includes nationally representative samples for all but one country in the sample (Nigeria). In Nigeria, the sample excluded some states due to security concerns but was representative at the state level for 12 of 37 states. In all countries, the team used a “multistage cluster-sampling design to ensure data were representative of rural and urban areas, private and public facilities, and of each facility type (e.g., front-line clinics versus hospitals).”_ Study size 10 Explain how the study size was arrived at _See the relevant text: “In total, the sample includes information from 89,826 vignettes from 13,754 providers and absenteeism observations from 22,747 providers across 8,061 facilities in 10 countries. In each country, the sample size was selected in order to provide representative data for the country and for sub-groups (e.g., rural versus urban areas).”_

- Quantitative variables 11 Explain how quantitative variables were handled in the analyses. If applicable, describe which groupings were chosen and why

_See the relevant text: “We present averages and 95% confidence intervals by first averaging all observations across facilities (or health workers) in each country (using country-specific facility weights to calculate a representative country average), and then averaging over the country means to calculate an overall average for the whole sample. (This is a suitable approach if we treat the country as the core unit of observation; i.e., we want to know how countries compare and how they perform as a group.)”_ 39
Statistical methods

- 12 (a) Describe all statistical methods, including those used to control for confounding

_As above. In this analysis, we are comparing averages: “This procedure is equivalent to calculating a weighted average and its standard error across all observations in the sample, where the weight on observation i in country j is_ i = ∏ i _, where_ i _denotes the (facility) weight of observation i in country j and is the sum of the facility weights_ kk≠ii kk ωω FF × ff ff _in country k.”_ kk rr rrkk FF = ∑ff (b) Describe any methods used to examine subgroups and interactions _See the relevant text: “Countries are divided according to performance. Figure S10 divides the countries into two groups based on performance. In the higher performance group, 49 percent of the health workers diagnose all or almost all of the conditions, compared to 11 percent in the lower performing group.”_ _“Diagnosis and prescription of correct treatment is also examined according to the type of disease. For some conditions, the key constraint appears to be at the diagnostic stage, as with diarrhea, where 59% of the providers diagnosed correctly but 91% of those who did so prescribed the correct treatment, and neonatal asphyxia, where 59% of the providers diagnosed correctly but 93% of those who did so prescribed the correct treatment. For other conditions, the key constraint instead appears to be prescribing the correct treatment, as with tuberculosis, where 81% of the providers correctly diagnosed the condition but only 37% of these providers prescribed the correct treatment, and post-partum hemorrhage, where 71% of the providers correctly diagnosed the condition but only 45% of them prescribed the correct treatment.”_ (c) Explain how missing data were addressed _The country-specific facility weights recreate a representative country average in the face of any missing facilities._ (d) If applicable, describe analytical methods taking account of sampling strategy _See the relevant text: We use “country-specific facility weights to calculate a representative country average.”_ <u>(e) Describe any sensitivity analyses</u> _In addition to the average analysis, we provide country-by-country and condition-by-condition analysis to demonstrate that our results are not driven by a single country or condition._

# **Results**

40
|Participants|13*|(a) Report numbers of individuals at each stage of study—eg numbers<br>potentially eligible, examined for eligibility, confirmed eligible, included in<br>the study, completing follow-up, and analysed<br>_See the relevant text: “The analysis in the study is at the level of the_<br>_provider. We used the full sample of providers available from the surveys._<br>_In total, the sample includes information from 89,826 vignettes from_<br>_13,754 providers and absenteeism observations from 22,747 providers_<br>_across 8,061 facilities in 10 countries.”_|
|---|---|---|
|||(b) Give reasons for non-participation at each stage<br>_The surveys were carried out in partnership with national governments and_<br>_so facilities participated._|
|||(c) Consider use of a flow diagram<br>_Not applicable._|
|Descriptive data|14*|(a) Give characteristics of study participants (eg demographic, clinical,<br>social) and information on exposures and potential confounders<br>_Because the paper covers samples in ten different countries, and descriptive_<br>_data on those samples are provided in country-specific studies, we provide_<br>_overarching characteristics of the country samples (e.g., in Table S1) as_<br>_compared to Africa as a whole._|
|||(b) Indicate number of participants with missing data for each variable of<br>interest|
|||_Not applicable, given that facilities participated in the surveys._|
|Outcome data|15*|Report numbers of outcome events or summary measures<br>_As discussed in the text, measures include (1) diagnostic accuracy, (2) rate_<br>_at which the correct prescription is given to the patient, (3) rate of provider_<br>_absence, (4) availability of drugs for treatment, (5) probability of care_<br>_readiness, namely (i) whether there is at least one provider working in the_<br>_clinic that can correctly diagnose and prescribe treatment for the_<br>_condition(s); (ii) health worker attendance; and (iii) whether or not the_<br>_facility has a minimum set of drugs to treat the condition(s)._|
|Main results|16|(a) Give unadjusted estimates and, if applicable, confounder-adjusted<br>estimates and their precision (eg, 95% confidence interval). Make clear<br>which confounders were adjusted for and why they were included<br>_The paper includes unadjusted estimates and confidence intervals: “(1)_<br>_Across all conditions and countries, health care providers correctly diagnose_<br>_64% (95% CI 0.62-0.65) of the cases. For diabetes, 35% (95% CI 0.33-0.37)_<br>_of the health workers provide a correct diagnosis (Figure 1A; Table S7). For_<br>_the three main killer diseases for children in Sub-Saharan Africa, accounting_<br>_for approximately one-third of all deaths among children under age 5 in the_<br>_countries surveyed, average diagnostic accuracy is 57% (95% CI 0.54-0.59)_|

41
_for pneumonia, 59% (95% CI 0.56-0.61) for diarrhea and 85% (95% CI 0.820.89) for malaria._ _(2) Diagnostic knowledge is associated with correct treatment, but a correct diagnosis is not a guarantee for prescribing correct treatment. The providers prescribed the correct (minimum) treatment following a correct diagnosis in 72% of the cases. Across all conditions and countries, health care providers were able both to correctly diagnose and treat 45% (95% CI 0.43-0.46) of the cases. For diarrhea, malaria, and neonatal asphyxia a majority of the providers managed to do so (54% (95% CI 0.43-0.46); 70% (95% CI 0.66-0.73); and 55% respectively). For the other four conditions, the majority of providers either fail to diagnose the condition correctly or fail to provide the correct treatment (Figure 1B; Table 2)._ _(3) The rate of provider absence is, on average, 30% (95% CI 0.29-0.32), with large variations across countries (see Table 2 and Table 4). For example, doctors in Togo and nurses in Uganda have absence rates of 50% (95% CI 0.26-0.74) and 47% (95% CI 0.43-0.50), respectively, while the average absence rate of doctors in Tanzania is 16% (95% CI 0.12-0.21)._ _(4) On average, 42% (95% CI 0.40-0.45) of the facilities were stocked with all four (or all three for Kenya) types of drugs and 70% (95% CI 0.69-0.73) of the facilities had a minimum set of functioning medical equipment (see Table S13). ORS is available in 84% (95% CI 0.82-0.87) of the facilities, ACT in 77% (95% CI 0.75-0.80) of the facilities, antibiotics in 69% (95% CI 0.660.72) of the facilities and oxytocin in 62% (95% CI 0.60-0.65) of the facilities (see Table 2)._ _(5) On average, only 14% (95% CI 0.12-0.15) of facilities are ready to provide selected child, neonatal, and maternity care that meets minimum quality standards, with (i) at least one provider available and able to correctly diagnose and treat diarrhea, pneumonia, post-partum hemorrhage and neonatal asphyxia, and (ii) minimum drugs as required (column (1), Table 5).”_ (b) Report category boundaries when continuous variables were categorized _Not applicable. The collected data includes only discrete variables. Any continuous variables are generated and are reported as continuous._ (c) If relevant, consider translating estimates of relative risk into absolute risk for a meaningful time period _Not relevant for this study._ Other analyses 17 Report other analyses done—eg analyses of subgroups and interactions, and sensitivity analyses _Countries are divided according to performance. Figure S11 divides the countries into two groups based on performance. In the higher performance group, 49 percent of the health workers diagnose all or_ 42
_almost all of the conditions, compared to 11 percent in the lower performing group._ _Diagnosis and prescription of correct treatment is also examined according to the type of disease. For some conditions, the key constraint appears to be at the diagnostic stage, as with diarrhea, where 59% of the providers diagnosed correctly but 91% of those who did so prescribed the correct treatment, and neonatal asphyxia, where 59% of the providers diagnosed correctly but 93% of those who did so prescribed the correct treatment. For other conditions, the key constraint instead appears to be prescribing the correct treatment, as with tuberculosis, where 81% of the providers correctly diagnosed the condition but only 37% of these providers prescribed the correct treatment, and post-partum hemorrhage, where 71% of the providers correctly diagnosed the condition but only 45% of them prescribed the correct treatment._

**<u>Discussion</u>** Key results 18 Summarise key results with reference to study objectives _See the relevant text: “This study offers a single metric that quantifies the relative contribution of staff knowledge, staff availability, and other facility inputs to the probability of care readiness which meets minimum quality standards.” “Our work shows severe gaps in care readiness in various countries in SubSaharan Africa, and health provider knowledge is a particularly severe constraint on the readiness of care across countries. In many facilities, providers are unable to diagnose accurately and propose appropriate treatment for conditions which account for major loss of life across the continent.”_ Limitations 19 Discuss limitations of the study, taking into account sources of potential bias or imprecision. Discuss both direction and magnitude of any potential bias _See the relevant text: “Not all misdiagnoses and mistreatments impose equal costs on patients and on health care systems. Some mistreatments – e.g., prescription of antibiotics for viral diarrhea – impose longer term costs on the patient population with greater resistance to antibiotics but limited short-term adverse impacts to the patient. In this analysis, we focus on the country-specific protocols, but one could imagine an analysis that weighs the direct and indirect cost to patients of different errors in diagnosis and treatment._

_Furthermore, there are times that deviations from diagnostic and treatment protocols may be guided by information rather than ignorance. In the case of prescribing antibiotics for diarrhea, recent evidence suggests that antibiotics promote growth among young children, so medical providers could be incorporating that information into their treatment. That said, while a medical provider may overprescribe antibiotics for that reason in practice, there is less reason to expect that she would do so in a_ 43
|||_vignette as a response to a specific diarrheal condition. In contexts where_<br>_providers seem to have limited skills, ensuring that guidelines adhere to the_<br>_best and latest knowledge and practice, and encouraging providers to_<br>_follow those guidelines, are likely to enhance the quality of care.”_|
|---|---|---|
|Interpretation|20|Give a cautious overall interpretation of results considering objectives,<br>limitations, multiplicity of analyses, results from similar studies, and other<br>relevant evidence<br>_See the relevant text: “Our findings highlight the need to adopt_<br>_competency-based pre-service and in-service training for health care_<br>_providers. As poor clinical knowledge is the key constraining factor at the_<br>_facility level, improving drug availability or reducing health workers’_<br>_absenteeism would only modestly increase the average care readiness that_<br>_meets minimum quality standards.”_|
|Generalisability|21|Discuss the generalisability (external validity) of the study results<br>_See the relevant text: “This study uses the lowest common denominator_<br>_across the country-specific protocols to define comparable care readiness_<br>_criteria across countries. Moreover, there is the well-known gap between_<br>_what health workers can do and what they actually do (the “can-do”_<br>_gap).22 For these reasons, we expect that in most surveyed countries the_<br>_probability of providing minimum quality care according to country-specific_<br>_guidelines is lower than that reported in this study.”_|
|**Other information**|||
|Funding|22|Give the source of funding and the role of the funders for the present<br>study and, if applicable, for the original study on which the present article<br>is based<br>_See the relevant text: “The funding is from the World Bank Service Delivery_<br>_Indicators Trust Fund (funded in large part by the Hewlett Foundation). The_<br>_funding source did not have any role in the design, data collection, data_<br>_analysis, data interpretation, or writing of the report.”_|

*Give information separately for exposed and unexposed groups.

**Note:** An Explanation and Elaboration article discusses each checklist item and gives methodological background and published examples of transparent reporting. The STROBE checklist is best used in conjunction with this article (freely available on the Web sites of PLoS Medicine at http://www.plosmedicine.org/, Annals of Internal Medicine at http://www.annals.org/, and Epidemiology at http://www.epidem.com/). Information on the STROBE Initiative is available at www.strobe-statement.org.

44
# **Table S2: How the Study Countries Compare with Sub-Saharan Africa as a Whole**

||**GDP per capita,**<br>**PPP (current**<br>**international $)**|**Life expectancy**<br>**at birth (years)**|**Maternal**<br>**mortality per**<br>**100,000 live births**<br>**(modeled**<br>**estimate)**|**Infant mortality**<br>**rate per 1,000**<br>**live births**|**Incidence of**<br>**malaria per**<br>**1,000 population**<br>**at risk**|
|---|---|---|---|---|---|
|Kenya|3,468|66|342|31|71|
|Madagascar|1,891|66|335|38|91|
|Mozambique|1,460|59|289|54|338|
|Niger|1,063|62|509|48|359|
|Nigeria|5,991|54|917|76|281|
|Senegal|3,783|67|315|32|65|
|Sierra Leone|1,602|54|1,120|79|380|
|Tanzania|3,227|64|524|38|113|
|Togo|1,774|60|396|47|371|
|Uganda|2,038|63|375|34|201|
|Average of ten<br>study countries|3,999|59|632|55|221|
|Average of Sub-<br>Saharan Africa|3,988|61|534|53|210|

Source: World Development Indicators 2019.<sup>1</sup> 45
# **Table S3. Disability adjusted life years lost to the diseases measured in the Service Delivery Indicators**

||**DALYs**<br>**(Disability-**<br>**Adjusted Life**<br>**Years, thousands)**|**YLDs**<br>**(Years Lived**<br>**with Disability,**<br>**thousands)**|**Prevalence**<br>**(thousands)**|**Incidence**<br>**(thousands)**|
|---|---|---|---|---|
|Diarrhoeal diseases|81,000|10,465|93,473|6,292,937|
|Diabetes|67,900|38,575|475,996|22,936|
|Neonatal encephalopathy<br>due to birth asphyxia and<br>trauma<br>_(closest in our study:_<br>_neonatal asphyxia)_|56,500|9,704|538,238|2,753|
|Tuberculosis|45,000|3,120|1,929,209|8,966|
|Malaria|45,000|1,468|136,085|208,768|
|Pneumococcal meningitis<br>_(closest in our study:_<br>_pneumonia)_|3,080|325|3,557|445|
|Maternal haemorrhage|2,230|61|1,660|6,988|

Source: Global Burden of Disease 2017.<sup>2,3</sup> 46
# **S4. Vignettes: Protocol, administration, reason for use, and validation**

A vignette is a hypothetical case in which the interviewer acts as a “patient” and provides a very brief description of symptoms. The provider is informed that the “patient” will comply with the provider’s instructions, medications, and tests. The provider is then invited to proceed exactly as he or she would under normal circumstances, asking questions about the history of the illness and performing necessary examinations and tests. The “patient” provides standardized predetermined answers to the questions and examination/test procedures. The questions, examinations, and tests that a provider uses are compared to protocol checklist. Data are recorded on each of the items on the checklist the provider provided; the diagnosis given; and the treatment suggested. Four of the seven vignettes used in this study were originally developed for use in Tanzania, with the items drawn from the national protocol.<sup>4</sup> The list was subsequently expanded to include items required by the guidelines for the Integrated Management of Childhood Illnesses.<sup>5,6</sup> Vignettes were administered to a random sample of doctors, medical assistants, nurses, and nurse-midwives (i.e., any staff that provide outpatient or maternity services) who were present in the facility on the day of the first, announced visit. Absenteeism on that first visit, announced in advance, was 17.2 percent (for nurses and midwives) and 25.4 percent (for doctors).

The assessments were carried out by a team of two enumerators who were trained in the administration of vignettes. One acted as the patient and the other recorded outcomes. Normally, the recorder should have medical training: for example, in Togo the enumerators were medical students and in Niger, enumerators were medical workers with at least a university degree.

We use data from seven vignettes capturing conditions associated with the main causes of child and maternal mortality in Sub-Saharan Africa: malaria, tuberculosis, diarrhea, pneumonia, diabetes, neonatal asphyxia and post-partum hemorrhage. All seven of these conditions are major global health challenges, with three of them only – diarrhea, neonatal asphyxia, and malaria – together causing an estimated 180,000 thousand disability-adjusted life years (see Table S3).

Table S5 below shows the starting scenario for the vignettes. Beyond the starting scenario, the vignettes proceed with additional information depending on the questions asked by the health provider.

**Table S5. Initial scenarios for vignettes**

|**Vignette**|**Description **|
|---|---|
|Acute diarrhea with severe<br>dehydration|An adult presents a 1 year child with diarrhoea.|
|Pneumonia|An adultpresent a5-yearold child withcough.|
|Diabetes Mellitus|An adult presents at the facility feeling weak<br>and without energy despite feeling hungry often<br>and eating frequently. The man is 48yrs old and<br>worksasaclerk.|
|Pulmonary Tuberculosis|A pregnant woman presents at the facility after<br>being told in the community<br>that she shouldgo for antenatal care.|
|Maleria + anemia|An adult presents a child with fever for some<br>time who hasgotten worse.|
|Post partum hemorrhage|A young woman presents at the facility with<br>vaginal bleeding 24 hours after delivery in a<br>health facility|

47
Neonatal Asphyxia A mother gives birth. The newborn is not crying. The newborn fails to establish regular breathing and appears pale and slightly blue.

There are a number of ways of scoring performance in patient case simulations, or vignettes. In this study we focus on diagnostic and treatment accuracy. We identified the lowest common denominators across the country-specific protocols to define comparable criteria across countries. This methodological choice implies that knowledge is assessed at a lower standard than more stringent WHO protocols.

The medical vignettes are used to measure clinic knowledge. That is, a health worker’s clinical vignette performance is not a measure of quality of care per se. It does, however, represent a good measure of the quality care which the provider can deliver, assuming he/she has access to relevant medical infrastructure. There are alternative methods to test for clinical knowledge and performance, including direct observation of real patient consultations and the use of standardized patients. Observation of real patient consultations, like vignettes, may be subject to observer effects (also referred to as Hawthorne effects). A study in the U.S. finds that the quality score derived using data from vignettes was similar to the quality score with standardized patients (i.e., trained actors who make unannounced visits to the clinic), but a study in India shows large gaps between vignettes and actual practice with standardized patients.<sup>7,8</sup> A study using data collected from 104 medical providers in northern Tanzania find that item by item, vignettes and direct observation produced identical results 63% of the time.<sup>8</sup> Aggregate scores derived from vignettes tend to be higher than scores from direct clinical observation, suggesting that knowledge measured using vignettes should be interpreted as an upper-bound of what providers could do; i.e., the vignettes measure what the medical provider knows, but not necessarily whether the provider applies this knowledge in practice.<sup>9,10</sup> The main advantages of using medical vignettes compared to direct observation of real patient consultations (or re-examination of patients outside of the clinic by well-qualified personnel), are that they are far less time-consuming and they increase comparability by avoiding issues related to differences in case mix. Moreover they provide a direct measure of providers’ clinical knowledge, rather than a combined measure of knowledge and effort. Data from medical vignettes is also directly policy relevant as the scores are closely linked to the quality of medical training.

Validation of vignettes typically involves comparing scores from vignettes with scores from direct observations.<sup>9,10</sup> This is less relevant for this study as we focus on provider knowledge, realizing that in practice (some) providers may not always fully utilize their knowledge when diagnosing and treating patients. Still, the vignettes utilized in this study has been (indirectly) validated as a testing method for provider knowledge. One source of validation is the fact that providers with a high share of relevant items performed are much more likely to suggest the correct diagnosis and treatment.<sup>9</sup> That is, better knowledge of which history taking question to ask and examinations to perform is highly correlated with both diagnostic and treatment accuracy. The two figures below, repeat that exercise using the full sample of vignettes and share of items performed (89,824 observations). We estimate a logit model with the binary response variable correct diagnosis [correct diagnosis and treatment] and with share of items performed as a continuous predictor and country and condition categorical predictors as control variables. The figures plot the adjusted probability of reaching the right diagnosis [right diagnosis and treatment]. As the share of items performed increases, the estimated probability of reaching a correct diagnosis and correct diagnosis followed by a correctly described treatment increase. The adjusted probability of correct diagnosis and treatment is estimated to be 33% for a 48
provider performing 10% of the items as compared to 94% for a provider performing 90% of the items. For a provider who asks no history taking questions nor perform any examinations and tests, the adjusted probability of correct diagnosis and treatment is below 25%.

<!-- Start of picture text --> S4A. Correct diagnosis S4B. Correct diagnosis and treatment<br>1 1<br>.8<br>.8<br>.6<br>.6<br>.4<br>.4 .2<br>0 .2 .4 .6 .8 1 0 .2 .4 .6 .8 1<br>Share of items performed Share of items performed<br>Predicted prob: Correct diagnosis<br>Predicted prob: Correct diagnosis & treatment<br><!-- End of picture text --> Note: Adjusted probability of reaching the right diagnosis (S4A) and right diagnosis and correctly described treatment (S4B) and 95% CI (shaded area). See text for details.

49
# **S6. Cross-country variation in treatment protocols for the vignettes**

In each country, the Service Delivery Indicators team asked the government for the official treatment guidelines for each treatment, then validated those along with the vignette in a workshop with a government-identified in-country doctor. The result is that there is crosscountry variation in treatment guidelines, but the variation is limited. To demonstrate, we provide the treatment guidelines listed in our data, resulting from the process above for postpartum hemorrhage, country by country. While there are minor variations across some countries, the general treatment protocols are very similar across countries.

|**Country**|**Treatmentprotocol**|
|---|---|
|Kenya|Determine the cause, uterine massage, oxytocin, prostaglandins, surgery if other<br>measures fail to stop bleeding, foley catheter, take blood grouping for cross<br>matching,referral, advice onothersigns|
|Madagascar|Determine the cause, uterine massage, IV line, oxytocin, IV infusion of plasmion,<br>blood transfusion, prostaglandins, misoprostol, uterine revision, surgery if other<br>measures fail to stop bleeding, foley catheter, referral|
|Mozambique|Uterine massage, Oxytocin, IV line, IV infusion of plasmion, blood transfusion,<br>prostaglandins, misoprostol, uterine revision, surgery if other measures fail to stop<br>bleeding, foley catheter, referral|
|Niger|Determine the cause, uterine massage, uterine revision, IV line, oxytocin,<br>ergometrine, IV infusion of plasmion, blood transfusion, progstaglandins,<br>misoprostol, surgery if other measures fail to stop bleeding, foley catheter, referral|
|Nigeria|IV line, take blood grouping for cross matching, foley catheter, uterine massage,<br>oxytocin, prostaglandins, surgeryifother measuresfailto stop bleeding,referral|
|Sierra Leone|Determine the cause, IV line, oxytocin, ergometrine, IV infusion of plasmion, blood<br>transfusion, prostaglandins, misoprostol, uterine revision, surgery if other measures<br>fail to stop bleeding, foley catheter, referral|
|Tanzania|Determine the cause, uterine massage, IV line, oxytocin, IV infusion of plasmion,<br>blood transfusion, prostaglandins, misoprostol, uterine revision, surgery if other<br>measures fail to stop bleeding, foley catheter, referral|
|Togo|Determine the cause, IV line, uterine massage, IV infusion of plasmion, blood<br>transfusion, take blood grouping for cross matching, foley catheter, uterine revision,<br>prostaglandins, surgeryifother measuresfailto stop bleeding,referral|
|Uganda|Determine the cause, IV line, uterine massage, take blood grouping for cross<br>matching, foley catheter, oxytocin, prostaglandins when available/misoprostol,<br>surgery, referral|
|Minimum<br>knowledge|Uterine massage and oxytocin|

50
# **Table S7. Minimum knowledge**

The minimum knowledge per vignette required for correct diagnose and correct treatment

|_Vignette_|_Correct diagnosis_|_Correct treatment_|
|---|---|---|
|Pneumonia|Pneumonia|Antibiotics|
|Diarrhea with severe<br>dehydration|Diarrhea with (severe) dehydration, or<br>acute diarrhea, or diarrhea|ORS or rehydration therapy|
|Tuberculosis|Tuberculosis|Combination of anti-<br>tuberculosis drugs|
|Malaria and anaemia|Malaria, or malaria and anemia, or<br>severe malaria|Artemisinin-based<br>combination therapy (ACT)|
|Neonatal asphyxia|Neonatal asphyxia or birth asphyxia|First aid assistance including<br>establishing open airway and<br>if need stimulate ventilation|
|Postpartum<br>hemorrhage|Postpartum hemorrhage|Uterine massage and oxytocin|
|Diabetes|Diabetes|Refer patient to a specialist|

51
# **S8. Selection of staff, drugs, and equipment for surveying**

During the first survey visit, information on all staff working at the facility, by professional cadre, and whether or not they were conducting patient consultations, was collected. Then at most 10 health workers were randomly sampled from the list of health workers present for the vignettes. Thus, the proportion of doctors, nurses, and midwives was not stratified and would roughly reflect the proportion of those professionals present at the clinic.

The World Health Organization Service Availability and Readiness Assessment (SARA) formed the basis for the list of drugs and equipment, intended to capture a list of essential or minimal items that should be available at every health facility.<sup>11</sup> Within each country, the SDI team collaborated with the government to adjust items on the list according to local guidelines and priorities. Availability of drugs and equipment was assessed during the first visit by visual inspections of storage facilities and consulting rooms at the health facilities.

A few days after the first visit, enumerators made a second unannounced visit to the facility and assessed the presence of a randomly pre-selected list of providers (or all providers, if less than 10 providers were available). Only staff on duty that day were included in the absenteeism measure. Staff who were away from the facility due to outreach or fieldwork were counted as present.

52
<!-- Start of picture text --> (6 )=9 [:- (1-p )<br><!-- End of picture text -->
# **Figure S10. Variation across the quality spectrum**

<!-- Start of picture text --> 1<br>.8<br>Kenya<br>Madagascar<br>.6 Nigeria<br>.4<br>.2<br>0<br>0 20 40 60 80 100<br>Percent of conditions diagnosed correctly (%)<br>cumulative probability<br><!-- End of picture text --> Notes: This figure shows the cumulative distributions of conditions diagnosed correctly for Kenya, Madagascar, and Nigeria. In Kenya, the curve is almost vertical at low scores, implying there are few providers that only manage to diagnose few conditions. In Nigeria and Madagascar, roughly 40 percent of the providers manage to diagnose 40 percent or less of the conditions, but Nigeria also has relative more providers that manage to diagnose most conditions; i.e., the slope of the curve is less steep in the case of Nigeria compared to Madagascar.

54
# **Figure S11. Clustering of Diagnostic and Treatment Accuracy**

# A. All countries

<!-- Start of picture text --> 60<br>50<br>40<br>30<br>20<br>10<br>0<br>0-1 conditions 2-3 conditions 4-5 conditions 6-7 conditions<br>B. Higher performing countries C. Lower performing countries<br>60 60<br>50 50<br>40 40<br>30 30<br>20 20<br>10 10<br>0 0<br>0-1 conditions 2-3 conditions 4-5 conditions 6-7 conditions 0-1 conditions 2-3 conditions 4-5 conditions 6-7 conditions<br>Share of providers<br>Share of providers Share of providers<br><!-- End of picture text --> Notes: The figures show the share (%) of providers who correctly diagnose at each number of vignettes. Panel A illustrates the results for all countries, while Panel B illustrates the results for Kenya, Mozambique, Sierra Leone, Tanzania, and Uganda, and Panel C the results for Madagascar, Niger, Nigeria, and Togo. All estimates are (unweighted) mean outcomes across countries, with the country means calculated using country-specific sampling weights. All providers were tested for 7 conditions using vignettes in Madagascar, Mozambique, Nigeria, Niger, Sierra Leone, Tanzania, Togo, and Uganda, and for 6 conditions (not malaria) in Kenya.

55
# **Table S12. Over prescription of antibiotics**

||(1)|(2)|(3)|
|---|---|---|---|
|Prescribe antibiotics<br>for diarrhea|Independent of<br>diagnosis|Incorrect<br>diagnosis|Correct<br>diagnosis|
|Kenya|0.40|0.49|0.37|
||(.38-.41)|(.45-.52)|(.35-.39)|
||[4505]|[1097]|[3408]|
|Madagascar|0.35|0.35|0.36|
||(.29-.42)|(.27-.43)|(.22-.50)|
||[642]|[513]|[129]|
|Mozambique|0.41|0.44|0.40|
||(37.-.46)|(.36-.52)|(.36-46)|
||[725]|[189]|[536]|
|Niger|0.17|0.18|0.17|
||(.12-.22)|(.04-.31)|(.11-.22)|
||[519]|[57]|[462]|
|Nigeria|0.48|0.44|0.60|
||(.46-.50)|(.41-.46)|(.57-.64)|
||[4549]|[3313]|[1236]|
|Senegal|0.20|0.27|0.16|
||(.10-.30)|(.11-.42)|(.05-.27)|
||[152]|[64]|[88]|
|Sierra Leone|0.32|0.47|0.31|
||(.28-.35)|(.30-.63)|(.27-.35)|
||[826]|[44]|[782]|
|Tanzania|0.30|0.28|0.30|
||(.24-.33)|(.09-.48)|(.24-.36)|
||[542]|[67]|[475]|
|Togo|0.42|0.47|0.18|
||(.29-.54)|(.32-.61)|(.05-.31)|
||[302]|[233]|[69]|
|Uganda|0.29|0.27|0.31|
||(.24-.33)|(.22-.33)|(.27-.35)|

56
||[709]|[422]|[287]|
|---|---|---|---|
|All|0.33|0.36|0.32|
||(.31-.36)|(.32-.40)|(.29-.35)|
||13,374|5,915|7,458|

Notes: The table reports the share of providers prescribe antibiotics for diarrhea, conditional on the diagnosis. All individual country statistics are calculated using country-specific sampling weights. 95% confidence intervals in parenthesis and number of observations in parenthesis.

57
**Table S13. Access to drugs and medical equipment by country**

||(1)|(2)|
|---|---|---|
||Drugs|Minimum medical<br>equipment|
|Kenya|0.83|0.85|
||(81-.85)|(.84-.87)|
||[1646]|[3064]|
|Madagascar|0.43|0.67|
||(.35-.52)|(.59-.75)|
||[459]|[459]|
|Mozambique|0.46|0.77|
||(.39-.53)|(.71-.83)|
||[204]|[204]|
|Niger|0.21|0.55|
||(.14-.27)|(.46-.64)|
||[256]|[256]|
|Nigeria|0.21|0.56|
||(.19-.23)|(.53-.58)|
||[2358]|[2366]|
|Senegal|NA|0.56|
|||(.39-.74)|
|||[151]|
|Sierra Leone|0.55|0.61|
||(.51-.60)|(.57-.66)|
||[544]|[544]|
|Tanzania|0.56|0.89|
||(.48-.64)|(.85-.93)|
||[399]|[399]|
|Togo|0.42|0.88|
||(.27-.56)|(.75-1.0)|
||[180]|[180]|
|Uganda|0.14|0.67|
||(.09-.20)|(.61-.73)|

58
||[390]|[390]|
|---|---|---|
|All|0.42|0.70|
||(.40-.45)|(.67-.73)|
||6,435|8,013|

Notes: The table reports the mean access to a set of essential drugs, and minimum set of medical equipment (by country), with all individual country statistics calculated using country-specific sampling weights. 95% confidence intervals in parenthesis and number of observations (facilities) in parenthesis. Drugs take the value 1 if the health facility is stocked with oral rehydration salts, antibiotics (amoxycillin or cotrimoxazole), artemisininbased combination therapy, and oxytocin, 0 otherwise. Medical equipment takes the value 1 if the health facility is equipped with a minimum set of functional medical equipment (thermometer, stethoscope, and sphygmomanometer), 0 otherwise.

59
# **Figure S14. Drugs and medical equipment at the facility**

<!-- Start of picture text --> Medical equipment<br>Oxytocin<br>Antibiotics<br>ACT<br>ORS<br>20 40 60 80 100<br>Facilities with drugs and medicial equipment (%)<br><!-- End of picture text --> Notes: ACT stands for artemisinin-based combination therapy, a treatment for malaria. ORS stands for oral rehydration salts, a treatment for diarrhea. Dots represent country-specific means, calculated using countryspecific sampling weights, vertical bars indicate median performance across countries, and boxes delineate the interquartile range (the middle 50 percent of values). Data are from visual inspections of storage facilities and consulting rooms at the health facilities from Kenya (2018), Madagascar (2016), Mozambique (2014), Nigeria (2013), Niger (2017), Senegal (2010), Sierra Leone (2018) Tanzania (2016), Togo (2014), and Uganda (2013). Medical equipment takes the value 1 if the health facility is equipped with a minimum set of functional medical equipment (thermometer, stethoscope, and sphygmomanometer), 0 otherwise. Drugs take the value 1 if the health facility is stocked with the drug, 0 otherwise. Data on availability of ACT was not collected for Kenya.

60
**Table S15. Probability of care that meets minimum quality standards (%): by country**

||(1)<br>|(2)<br>Probability of<br>|(3)<br>Care Readiness<br>|(4)<br>|
|---|---|---|---|---|
|_Condition_|As observed|Assume<br>available<br>drugs|Assume<br>provider<br>presence|No. of<br>facilities|
|Kenya|45.3|51.2|60.2|1,646|
||(43.0-47.6)|(49.0-53.5)|(57.7-62.7)||
|Madagascar|1.9|2.0|1.9|459|
||(-0.3-4.0)|(-0.2-4.1)|(-0.3-4.0)||
|Mozambique|16.4|19.6|17.2|204|
||(11.5-21.4)|(14.3-24.9)|(12.0-22.4)||
|Niger|4.3|8.2|5.8|256|
||(2.0-6.6)|(4.4-11.9)|(2.8-8.8)||
|Nigeria|3.8|6.4|4.3|2,358|
||(2.8-4.8)|(5.2-7.6)|(3.2-5.4)||
|Sierra Leone|25.5|40.6|31.0|399|
||(21.6-29.5)|(36.4-44.8)|(26.7-35.4)||
|Tanzania|25.4|39.1|27.4|180|
||(20.1-30.7)|(32.4-45.9)|(21.8-33.1)||
|Togo|0.0|0.0|0.0|390|
|Uganda|-<br>1.0|-<br>4.7|-<br>1.3|544|
||(-0.1-2.1)|(2.2-7.2)|(0.1-2.6)||

**Note** : Column (1) reports the estimated probability of care readiness that meets minimum quality standards for child, neonatal & maternity care (diarrhea, pneumonia, post-partum hemorrhage, and neonatal asphyxia). Column 2 reports the estimated probability of care readiness that meets minimum quality standards, assuming essential drug(s) (oral rehydration salts or rehydration therapy for diarrhea; antibiotics for pneumonia; and oxytocin for postpartum hemorrhage) for treating the conditions are available. No essential drugs data were collected for neonatal asphyxia. Column 3 reports the estimated probability of care readiness that meets minimum quality standards, assuming no more than 5% absenteeism. All individual country statistics calculated using country-specific sampling weights. 95% confidence intervals in parenthesis. Number of facilities reported in column 4.

61
# **Figure S16. Comparing estimates of minimum quality care conditional on drugs availability and low absence.**

<!-- Start of picture text --> A1. Diarrhea: Drugs available B1. Diarrhea: Low absence<br>100 100<br>TZA TZA<br>MOZ<br>SLE MOZ KEN SLE<br>80 80<br>NER<br>NER<br>KEN<br>60 60<br>UGA<br>40 40<br>UGA<br>NGA<br>NGA<br>MDG MDG<br>20 20 TGO<br>TGO<br>0 0<br>0 20 40 60 80 100 0 20 40 60 80 100<br>Pr. of min. quality standards (%) Pr. of min. quality standards (%)<br>A2. Pneumonia: Drugs available B2. Pneumonia: Low absence<br>100 100<br>90 90<br>TZA<br>80 80 KEN<br>TZA<br>70 70<br>KEN<br>MOZ<br>60 60<br>SLE SLE MOZ<br>50 50 TGO<br>40 NGA TGO 40<br>NER MDG MDG<br>UGA<br>30 30 NER<br>NGA<br>20 20 UGA<br>10 10<br>0 0<br>0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100<br>Pr. of min. quality standards (%) Pr. of min. quality standards (%)<br>A3. Malaria: Drugs available B3. Malaria: Low absence<br>100 TZA 100 SLE<br>MOZ TZA<br>SLE<br>MDG UGA<br>80 80<br>NER MDG<br>NGA UGA<br>TGO<br>60 60<br>TGO NERMOZ<br>NGA<br>40 40<br>20 20<br>0 0<br>0 20 40 60 80 100 0 20 40 60 80 100<br>Pr. of min. quality standards (%) Pr. of min. quality standards (%)<br>Pr. of min. quality standards: No absence (%)<br>Pr. of min. quality standards: Drugs available (%)<br>Pr. of min. quality standards: No absence (%)<br>Pr. of min. quality standards: Drugs available (%)<br>Pr. of min. quality standards: No absence (%)<br>Pr. of min. quality standards: Drugs available (%)<br><!-- End of picture text --> 62
A4. Post-partum hemorrhage: Drugs available B4. Post-partum hemorrhage: Low absence <!-- Start of picture text --> 100 100<br>KEN<br>80 SLE KEN 80<br>SLE<br>TZA<br>60 60<br>MOZ MOZTZA<br>40 40<br>UGA<br>NGAMDG UGAMDG<br>20 NER 20 NGA<br>NER<br>TGO TGO<br>0 0<br>0 20 40 60 80 100 0 20 40 60 80 100<br>Pr. of min. quality standards (%) Pr. of min. quality standards (%)<br>A5. Child care: Drugs available B5. Child: Low absence<br>100 100<br>TZA<br>80 80<br>TZA<br>60 MOZ 60 KEN<br>SLE KEN MOZ<br>SLE<br>40 40<br>NER<br>NER<br>20 NGA 20<br>TGONGA<br>UGATGOMDG MDG<br>UGA<br>0 0<br>0 20 40 60 80 100 0 20 40 60 80 100<br>Pr. of min. quality standards (%) Pr. of min. quality standards (%)<br>A6. Neonatal and maternal care: Drugs<br>available B6. Neonatal and maternal care: Low absence<br>100 100<br>KEN<br>80 SLE 80<br>KEN<br>SLE<br>60 60<br>TZA<br>40 MOZ 40 MOZTZA<br>UGA<br>20 NGAMDG 20 UGAMDG<br>NGA<br>NER<br>NER<br>0 TGO 0 TGO<br>0 20 40 60 80 100 0 20 40 60 80 100<br>Pr. of min. quality standards (%) Pr. of min. quality standards (%)<br>Pr. of min. quality standards: No absence (%)<br>Pr. of min. quality standards: Drugs available (%)<br>Pr. of min. quality standards: No absence (%)<br>Pr. of min. quality standards: Drugs available (%)<br>Pr. of min. quality standards: No absence (%)<br>Pr. of min. quality standards: Drugs available (%)<br><!-- End of picture text --> available B6. Neonatal and maternal care: Low absence

**Note** : Panel A plots the relationship between the probability of care readiness that meets minimum quality standards vs. the probability of care readiness that meets minimum quality standards assuming essential drugs treating the conditions are available. Panel B plots the relationship between the probability of care readiness that meets minimum quality standards, vs. the probability of care readiness that meets minimum quality standards

63
assuming all providers are present at least 95% of the time. Countries that are higher than the 45 degree line are countries where our estimates suggest that having drugs available (Panel A) and reducing absenteeism (Panel B) would significantly improve the probability of care readiness that meets minimum quality standards. Essential drugs are ORS or rehydration therapy for diarrhea; antibiotics for pneumonia; ACT for malaria; and oxytocin for postpartum hemorrhage. Child care includes two conditions (diarrhea, pneumonia). Neonatal & maternal care includes two conditions (post-partum hemorrhage and neonatal asphyxia). Data are from clinical vignettes, unannounced visits, and visual inspections from Kenya (2018), Madagascar (2016), Mozambique (2014), Nigeria (2013), Niger (2017), Sierra Leone (2018), Tanzania (2016), Togo (2014), and Uganda (2013). ISO 3-digit alphabetic codes are: KEN (Kenya), MDG (Madagascar), MOZ (Mozambique), NER (Niger), NGA (Nigeria), SLE (Sierra Leone), TZA (Tanzania), TGO (Togo), UGA (Uganda).

64
# **Figure S17. Share of providers who provide accurate diagnosis and treatment conditional on availability of equipment and drugs**

# A. Medical equipment

<!-- Start of picture text --> 40<br>30<br>20<br>10<br>0<br>Facilities without equipment Facilities with equipment<br>No condition One condition Both conditions<br>Share of providers<br><!-- End of picture text -->

**Note** : Mean number of conditions (0,1,2) diagnosed and treated correctly by the most knowledgeable provider in the facility conditional on the facility having access or not to a minimum set of functional medical equipment (thermometer, stethoscope, and sphygmomanometer). Two conditions: diarrhea and pneumonia. The estimates are (unweighted) mean outcomes across countries, with the country means calculated using country-specific sampling weights. The error bars represent the 95-percent confidence interval. Data are from clinical vignettes and visual inspections of storage facilities and consulting rooms from Kenya (2018), Madagascar (2016), Mozambique (2014), Nigeria (2013), Niger (2017), Senegal (2010), Sierra Leone (2018) Tanzania (2016), Togo (2014), and Uganda (2013).

# B. Antibiotics

<!-- Start of picture text --> 60<br>50<br>40<br>30<br>20<br>10<br>0<br>Facilities without antibiotics in stock Facilities with antibiotics in stock<br>Cannot manage condition Can manage condition<br>Share of providers<br><!-- End of picture text -->

**Note** : Share of the most knowledgeable provider in the facility that can provide correct diagnosis and treatment for pneumonia, conditional on the facility having access or not to antibiotics. The estimates are (unweighted) mean outcomes across countries, with the country means calculated using country-specific sampling weights. The error bars represent the 95-percent confidence interval. Data are from clinical vignettes and visual inspections of storage facilities and consulting rooms from Kenya (2018), Madagascar (2016), Mozambique (2014), Nigeria (2013), Niger (2017), Sierra Leone (2018) Tanzania (2016), Togo (2014), and Uganda (2013).

65
# C. Oral rehydration salts (ORS)

<!-- Start of picture text --> 80<br>70<br>60<br>50<br>40<br>30<br>20<br>10<br>0<br>Facilities without ORS in stock Facilities with ORS in stock<br>Cannot manage condition Can manage condition<br>Share of providers<br><!-- End of picture text -->

**Note** : Share of the most knowledgeable provider in the facility that can provide correct diagnosis and treatment for pneumonia, conditional on the facility having access or not to antibiotics. The estimates are (unweighted) mean outcomes across countries, with the country means calculated using country-specific sampling weights. The error bars represent the 95-percent confidence interval. Data are from clinical vignettes and visual inspections of storage facilities and consulting rooms from Kenya (2018), Madagascar (2016), Mozambique (2014), Nigeria (2013), Niger (2017), Sierra Leone (2018) Tanzania (2016), Togo (2014), and Uganda (2013).

66
# **Supplementary Appendix References**

- 1 World Bank. World Development Indicators. 2019. https://databank.worldbank.org/source/world-development-indicators# (accessed 3 Mar 2020).

- 2 Global Burden of Disease 2017 DALYs and HALE Collaborators. Global, regional, and national disability-adjusted life-years (DALYs) for 359 diseases and injuries and healthy life expectancy (HALE) for 195 countries and territories, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017 - ScienceDirect. _Lancet_ 2018; **392** :1859–922.

- 3 Global Burden of Disease 2017 Disease and Injury Incidence and Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990–2017: a systematic analysis for the Global Burden of Disease Study 2017 - ScienceDirect. _Lancet_ 2018; **392** :1789–858.

- 4 Leonard KL, Masatu MC. The use of direct clinician observation and vignettes for health services quality evaluation in developing countries. _Social Science & Medicine_ 2005; **61** :1944–51. doi:10.1016/j.socscimed.2005.03.043

- 5 Bold T, Svensson J, Gauthier B, _et al._ Service delivery indicators: Pilot in education and health care in Africa. Bergen, Norway: : Chr. Michelsen Institute 2011.

- 6 World Health Organization U. _Model IMCI handbook: Integrated management of childhood illness_ . 2005.

- https://www.who.int/maternal_child_adolescent/documents/9241546441/en/

- 7 Peabody JW, Luck J, Glassman P, _et al._ Comparison of Vignettes, Standardized Patients, and Chart Abstraction: A Prospective Validation Study of 3 Methods for Measuring Quality. _JAMA_ 2000; **283** :1715–22. doi:10.1001/jama.283.13.1715

- 8 Mohanan M, Vera-Hernández M, Das V, et al. The know-do gap in quality of health care for childhood diarrhea and pneumonia in rural India. JAMA pediatrics Published Online First: 2015. doi:10.1001/jamapediatrics.2014.3445

- 9 Leonard KL, Masatu MC. Variations in the Quality of Care Accessible to Rural Communities in Tanzania. _Health Affairs_ 2007; **26** :w380–92. doi:10.1377/hlthaff.26.3.w380

- 10  Das J, Hammer J, Leonard K. The Quality of Medical Advice in Low-Income Countries. _Journal of Economic Perspectives_ 2008; **22** :93–114. doi:10.1257/jep.22.2.93

- 11  World Health Organization. Service availability and readiness assessment (SARA). 2015. http://www.who.int/healthinfo/systems/sara_introduction/en/

67