metadata: authors: 0: author: Das, Jishnu
1: author: Hammer, Jeffrey
2: author: Leonard, Kenneth
count: India,Indonesia,Tanzania,Paraguay
docna: 0: docna: The quality of medical advice in low-income countries
docty: Policy Research Working Paper
subtopic: Pharmaceuticals Industry,Health Care Services Industry,Labor Markets,Pharmaceuticals & Pharmacoeconomics,Cholera,Malaria,Tuberculosis,Communicable Diseases,Leprosy
teratopic: Health, Nutrition and Population,Gender
abstracts: cdata!: This paper provides an overview of
            recent work on quality measurement of medical care and its
            correlates in four low and middle-income countries-India,
            Indonesia, Tanzania, and Paraguay. The authors describe two
            methods-testing doctors and watching doctors-that are
            relatively easy to implement and yield important insights
            about the nature of medical care in these countries. The
            paper discusses the properties of these measures, their
            correlates, and how they may be used to evaluate policy
            changes. Finally, the authors outline an agenda for further
            research and measurement.
geo_reg_key: 80469,80451,80470,80475,80471,80458,80473,80450
repnme: repnme: The quality of medical advice in low-income countries
display_title: The quality of medical advice in
            low-income countries
listing_relative_url: /research/2008/01/8978038/quality-medical-advice-low-income-countries
topicv3: Tuberculosis m1326131 1419,Malaria m1327042 1420,Pharmaceuticals m1327740 1350,Jobs m1503931 2670,Health m1504661 3480
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Health, Nutrition and Population,Gender
no_of_pages: 38
author_keys: 0: author_key: NA
1: author_key: NA
2: author_key: NA
model_extractions: WPS4501

# Policy ReseaRch WoRking PaPeR 4501

The Quality of Medical Advice in Low-Income Countries _Jishnu Das Jeffrey Hammer Kenneth Leonard_ The World Bank Development Research Group Human Development and Public Services Team January 2008
Policy ReseaRch WoRking PaPeR 4501

# **Abstract**

This paper provides an overview of recent work on quality measurement of medical care and its correlates in four low and middle-income countries—India, Indonesia, Tanzania, and Paraguay. The authors describe two methods—testing doctors and watching doctors— that are relatively easy to implement and yield important insights about the nature of medical care in these countries. The paper discusses the properties of these measures, their correlates, and how they may be used to evaluate policy changes. Finally, the authors outline an agenda for further research and measurement.

This paper—a product of the Human Development and Public Services Team, Development Research Group—is part of a larger effort in the department to analyze the quality of medical care in low-income countries. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at jdas1@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
**The Quality of Medical Advice in Low-Income Countries Jishnu Das, Jeffrey Hammer, and Kenneth Leonard**<sup>**1**</sup>

> 1 Jishnu Das is a Senior Economist, Development Research Group, World Bank, Washington, D.C., and Visiting Fellow, Center for Policy Research, New Delhi, India. Jeffrey Hammer is Charles and Marie Robertson Visiting Professor in Economic Development, Woodrow Wilson School, Princeton University. Kenneth Leonard is an Assistant Professor, Department of Agricultural and Resource Economics, University of Maryland, College Park, Maryland. Das is the corresponding author at <jdas1@worldbank.org>.
# **Introduction**

In 1978, an International Conference on Primary Health Care was held at Alma Ata in what was then the USSR, and is now Kazakhstan. The resulting Alma Ata Declaration called _“for urgent action by all governments, all health and development workers, and the world community to protect and promote the health of all the people of the world_ ” (World Health Organization, 1978). While “action” was understood to mean a wide variety of interventions including safe water, sanitation, nutrition and pest control, primary health **_care_** was also emphasized and, in the event, captured a larger proportion of health budgets than purely preventive services.

Since then, increased investment in health care infrastructure in many low-income countries around the world means that urban and rural households have improved access to health facilities and doctors. While the availability of health care remains an issue in certain areas, like some countries in Sub-Saharan Africa, for a large majority of households in lowincome countries, problems of access are no longer the first order concern that they were in 1978.

Whether or not this gain in health infrastructure led to improved health, it is clear that in low-income countries today, access to health care facilities and personnel often do not translate into health. Two case-studies help understand why.

Ms. Sundar is a typical patient who lives in urban Delhi. There are over 70 private-sector medical care providers within a 15- minute walk from her house (and virtually any household in her city). She chooses the private clinic run by Dr. SM and his wife. Above the clinic a prominent sign says “ _Ms. MM, Gold Medalist, MBBS_ ”, suggesting that the clinic is staffed by a highly proficient doctor (an MBBS is the basic degree for a medical doctor as in the British 2
system). As it turns out, Ms. MM is rarely at the clinic. We were told that she sometimes comes at 4 a.m. to avoid the long lines that form if people know she is there. We later discover that she has “franchised” her name to a number of different clinics.

Therefore, Ms. Sundar sees Dr. SM and his wife, both of whom were trained in traditional Ayurvedic medicine through a six-month long-distance course. The doctor and his wife sit at a small table surrounded, on one side, by a large number of bottles full of pills, and on the other, a bench with patients on them, which extends into the street. Ms. Sundar sits at the end of this bench. Dr. SM and his wife are the most popular medical care providers in the neighborhood, with more than 200 patients every day. The doctor spends an average of 3.5 minutes with each patient, asks 3.2 questions, and performs an average of 2.5 examinations. Following the diagnosis, the doctor takes two or three different pills, crushes them using a mortar and pestle, and makes small paper packets from the resulting powder which he gives to Ms. Sundar and asks her to take for two or three days. These medicines usually include one antibiotic and one analgesic and anti-inflammatory drug. Dr. SM tells us that he constantly faces unrealistic patient expectations, both because of the high volume of patients and their demands for treatments that even Dr. SM knows are inappropriate. Dr. SM and his wife seem highly motivated to provide care to their patients and even with a very crowded consultation room they spend more time with their patients than a public sector doctor would. However, they are not bound by their knowledge of health care and instead deliver the health care, like the crushed pills in a paper packet, which will result in more patients willing to pay more for their services. Indeed, over-medication in India is a widespread (for instance, Greenhalgh, 1987; Phadke, 1998). Note, this is consumer driven and not “supplier induced demand” of practitioners exploiting asymmetric information to talk people into unnecessary treatment.

3
In rural Tanzania, Ms M, brings her 9 month old to the local health clinic, carrying the child on her back. When she enters, Dr. K (an Assistant Medical Officer with O-level education and 4 years of medical training) asks her what the problem is. Still standing in front of his desk, she replies that her daughter has a fever. Dr. K fills a prescription for malaria based on this statement, even though he cannot see the child, much less observe her condition. The consultation and medicine are both free and Ms. M leaves the facility with the prescribed medicine. During the exit interview a nurse on our team notes that the child is suffering from severe pneumonia. The health facility has the medicine to treat both malaria and pneumonia. Dr. K is trained in the diagnosis and treatment for these diseases and saw only 25 patients that day. Yet, but for the intervention of the nurse on our research team, the child would have died. Indeed, a survey in rural Tanzania found that 79 percent of children who die of malaria sought care at modern health facilities (de Savigny et al. 2004).

Taking these two cases as a starting point, this paper documents the quality of medical advice in low-income countries. We being with an overview of the evidence that in many lowincome countries, the problem of access to health care has improved, at least if measured by facilities, doctor visits, and access to drugs. We then introduce a rapidly developing literature on measuring the quality of health care in low income countries. Two broad approaches have been used. In the “vignette” approach, a medical provider may be presented with hypothetical cases, and their responses may be compared to a checklist of essential procedures. The other approach is direct observation of doctors.

These two tools that measure process quality—medical vignettes and direct observation of the doctor-patient interaction—have proved quite informative. For example, doctors in Tanzania complete less than a quarter of the essential checklist for patients with classic 4
symptoms of malaria, a disease that kills 63,000-96,000 Tanzanians each year. A public sector doctor in India asks one (and only one) question in the average interaction—“What’s wrong with you?” In Paraguay, the amount of time a doctor spends with a patient has nothing to do with the severity of the patient’s illness. We present systematic evidence in the paper to show that these isolated facts represent common patterns. We interpret the vignettes as a measure of what health care providers know, while direct observation shows what they actually do.

By looking at what doctors know and what they do, we are able to show four things. First, the quality of care in low-income countries as measured by what doctors know is very low. Second, the problem of low competence is compounded due to low effort—doctors provide lower standards of care for their patients than they know how to provide. Third, the poor are particularly disadvantaged both because they have access only to doctors with low level levels of competence and because less competent doctors put in less effort. Fourth, oft-tried measures such as training doctors are unlikely to work—because doctors exert very little effort, three years of medical school in Tanzania results in only a 1 percentage point increase in the probability of a correct diagnosis.

Our evidence on health care quality in low-income countries is drawn primarily from studies in four countries: Tanzania, India, Indonesia and Paraguay. Table 1 summarizes the methods used and the samples in these four countries. These results for these four countries must obviously be viewed as preliminary, and in the conclusion of the paper, we will outline some of the important theoretical and empirical questions yet to be investigated. However, the available evidence strongly suggests that if the enormous burden that poor health imposes on most of he world’s poor is to be reduced through medical care, it will be critical to look beyond the presence (or absence) of physical components of health care like proximity to facilities and to examine the 5
behavior of health care providers. An alternative interpretation is that the public sector may have serious, systematic, problems ensuring quality in publicly supplied medical care and that major improvements in true public goods and non-medical control of infectious disease are needed before turning to policies that are hard to implement.

# **Improved Access to Health Care in Low-Income Countries**

The expansion of physical health care infrastructure has improved the extent to which people in low-income countries can meet with a health care provider. In 2004 data from the Demographic and Health Surveys show that 68 percent of urban and 58 percent of rural Tanzanian reported that they took their child to a health facility when he or she showed signs of acute respiratory infection (viral or bacterial colds and coughs). In India, for the same period, 78 percent of urban and 60 percent of rural residents report having done so (National Family Health Survey-3 data for 2005-06). In Indonesia, the overall rate is 62 percent, and in Paraguay in 1990 the rate was 53 percent. Although, there is no precise comparison in the U.S., data from the National Medical Expenditure Survey (1988) show that 52 percent of children seek care at a health facility when they are sick with pharyngitis (throat infection); among uninsured the rate is 32 percent. The comparability of these numbers to US data, in conjunction with the fact that the rural health facility usage rates are within 80 percent of the urban rates, suggests that most residents of low-income countries can access health care when they need to.

Detailed surveys on health care utilization in low-income countries also often show many contacts with health care providers. People in rural Rajasthan—a low-income and low-density state in India--visit a doctor about six times a year (Banerjee, Deaton and Duflo, 2004). In urban India, individuals visited doctors about five times a year (Das and Sanchez-Paramo, 2003). This 6
usage of health facilities in India—both in a relatively rich urban sample and a relatively poor rural sample—is higher than the U.S. average of 3.15 visits per person per year (NAMS advance data available at http://www.cdc.gov/nchs/data/ad/ad374.pdf). Because medicines that typically require prescriptions in the U.S. (such as antibiotics) are usually sold over-the-counter in India, it is likely that under similar prescription regimes, the differences in care seeking would be even higher. Other low-income countries have similar results. Across countries ranging from Burkina Faso (whose Gross National Income or GNI per capita was $230 in 1998) to Thailand (GNI per capita $2,740 in 1998), health care utilization is high among the population in general, and among the poorest quintile in particular (Mackinnon et al., 2000).

That patients in low-income countries can access facilities is encouraging. However, whether the higher usage of health facilities translates into better health is a matter of debate. In many countries, the level of expertise of medical care providers is low. In China, the use of barefoot doctors was widely advocated and used – although ironically, this program was abandoned in China just as the rest of the world began to emulate it. In India, the government provides valid licenses to a wide range of health care practitioners: from “real” doctors with a MBBS to a “Practicing Medical Physician” or PMP, whose only qualification is that the person has practiced medicine at some time. In addition, India’s government has brought a large number of semi-trained medical care providers into the medical practice, and “regularized” the use of alternative methods of treatment. African countries trained new cadres of clinicians that required very little formal medical education. In Tanzania, with four medical doctors for every 100,000 people, much care in the rural areas is provided by clinical officers (who have four years of secondary schooling and two additional years of medical training) and clinical assistants (who have only elementary education and three years of medical training).

7
Health outcomes in these countries are still very poor compared to the United States _despite_ very high and frequent use of the health care system by the population. One possibility is that poverty and the overall health environment make good medical care insufficient for good health. A second possibility is that the expansion of the health care system has led to a situation where the poor have access to doctors, but the quality of the medical care is so poor that even common illnesses are not diagnosed and treated correctly. Almost certainly, both factors play a role. To ascertain how important each might be, an important first step is to measure and document the quality of medical advice and its correlates in these countries.

# **Competence: What Do We Learn From Testing Doctors?**

Traditionally, health care quality in low-income countries has been measured either by the condition of infrastructure or by the presence or absence of certain drugs—measures known in the literature as “structural quality”. While clearly not a measure of the quality of medical advice, structural quality may be a good proxy for the quality of medical advice. This line of reasoning runs into two problems. First, in a number of countries, there is little variation in basic measures of health infrastructure. In Paraguay and Indonesia, for instance, more than 95 percent of all facilities have running water, regular electricity, and telephone connections and just less than 90 percent have refrigeration facilities, important for maintaining vaccination cold-chains. Second, even when there is variation in structural quality (as there would be for the availability of medicines), in both Indonesia and Tanzania there is little or no correlation with the quality of medical advice that is actually dispensed in these facilities (Barber, Gertler and Harimurti 2007). An alternative approach to measuring the quality of medical care is by testing doctors using vignettes. A vignette is a hypothetical case in which the interviewer acts as a patient and 8
provides a very brief description of symptoms. In addition, the doctor is told that the patient will comply with the provider’s instructions, medications and tests, and will follow-up by returning to the doctor if necessary. The doctor is then invited to proceed exactly as he or she would under normal circumstances, asking questions about the history of the illness and performing necessary examinations. The “patient” provides standardized pre-determined answers to the questions and examination procedures. Usually, a second interviewer is present to provide answers to questions that the patients may not know, to provide continuity to the process—for example, if the results of a blood-test are called for, the second interviewer responds. The recorder also notes the treatment described.

# _An Example of a Vignette_

In one vignette, used in India, Indonesia and Tanzania, a mother brings an infant to the provider. She tells the provider that her child has been suffering from diarrhea for the last two days and she does not know what to do. The recorder notes such things as whether the doctor asked about the nature of the stool (to distinguish viral diarrhea from dysentery) and checked the child for symptoms of severe dehydration, such as dryness of the tear-ducts or depression in the skull fontanel (which are the soft spots in the skull of an infant). The underlying fact, reflected in answers given, is that the diarrhea does not arise from an infection and that the child is not severely dehydrated, so that oral re-hydration therapy is the only required action.

The questions and examinations that a clinician uses are compared to a list of questions and examinations compiled by experts (India) or drawn from national protocol (Tanzania). Thus, for each clinician evaluated, and for each illness presented in a vignette, data are collected on whether the clinician provided each of the items on the reference list, the diagnosis given and the 9
treatment suggested. In this manner, each doctor is tested for up to six different illnesses. The performance of each clinician on the list of items is then used to create an index of overall competence. In addition, the diagnoses and treatments offered can be evaluated for correctness, either by comparison with a protocol or by using a team of medical experts.

# _Vignettes: Results and Validity_

Figure 1 presents a snapshot of vignette responses India, Indonesia and Tanzania. Although the vignettes were independently designed in these countries, all three administered a vignette on childhood diarrhea; comparisons of whether certain diagnosis questions were asked are shown in the top panel. The projects in India and Indonesia also administered a vignette on tuberculosis, and the results for some diagnosis questions are shown in the bottom panel. The percentage of clinicians who asked each question is shown by quintiles of the competence index (more on this below), derived from performance on all items across all vignettes administered.

The most noteworthy result is that basic essential procedures related to common diseases are not used by health care providers. In India, where close to 500,000 children die every year of diarrhea, only 25 percent of providers in the richest state, Delhi, asked about blood/mucous in the stool, 49 percent ask whether the child has a fever, and 7 percent checked for a depression in the skull fontanel. These essential questions and examinations allow the provider to differentiate viral from bacterial causes and to assess the degree of dehydration—thus, whether the child needs immediate hospitalization. In Tanzania, these numbers are only slightly better, and even in Indonesia, except for asking about the frequency of stools, there is still less than a 50 percent chance that any of the other essential procedures would be requested.

10
The results also show that the probability of asking an essential question or completing an essential examination increases with competence, sometimes dramatically so. Across the different questions, moving from the lowest to the highest quintile of competence increases the probability of completing an essential procedure by 40-70 percentage points. Competence derived from the vignettes is strongly associated with greater knowledge of essential procedures. Also noteworthy is that in India, the probability of completing essential procedures does _not_ vary much across the bottom two quintiles—this was because the vignettes were “too difficult” for doctors in the bottom part of the distribution—something that is discussed in Das and Hammer (2005).

Comparing the diarrhea and the TB panel also shows why comparisons across countries will be difficult. Ideally, competence would not vary across different procedures, and this is indeed what we see in any comparison within each of the three countries. However, comparing India and Indonesia, doctors who are competent in one dimension may be less competent in another. For example, providers in India look much worse than their Indonesian counterparts for diarrhea, where the difference in the probability of completing an essential procedure between the two countries is close to 40 percentage points. For Tuberculosis the difference is again close to 40 percentage points in favor of Indonesia for whether the providers complete a chest examination, but Indian providers are far more likely to ask the patients for a sputum examination—a recommended and critical procedure, without which a diagnosis of Tuberculosis cannot be confirmed. Doctors in India come across looking more or less competent depending upon which procedure we are looking at, which may reflect differences in the disease burden or the medical training across these two countries.

11
# _Constructing and Validating the Competence Index_

Extending the discussion of Figure 1, Item Response Theory (a tool of psychometrics) solves for optimal weights for each question in creating a composite index. The methodology is sufficiently flexible to allow some questions to have a high weight at the low end of the competence distribution, and others to have a high weight at the higher end of the competence distribution. These weights allow for a competence measure that provides the greatest possible information about the differences between clinicians, and simultaneously allows us to verify that the index of competence explains behavior.<sup>2</sup> One source of validation for the competence index is the fact that doctors with a high value of the index are much less likely to suggest harmful treatment (India) and much more likely to suggest the correct diagnosis and treatment (Tanzania). As another indicator of how poor overall competence is, a doctor in India has to be above mean competence in the sample to have a better than even chance of not harming the patient.

A second indication of the importance of this index is that, in Tanzania and India, after controlling for a doctor’s score on the vignette, no other characteristics of the doctor are significantly correlated with the probability of giving the correct diagnosis, suggesting that the scores derived on the vignette provides a useful summary of knowledge and/or capacity (Das and Hammer, 2006; Leonard and Masatu, 2007).

_What Correlates with Competence?_

> 2 The critical assumption for constructing the competence index is uni-dimensionality. That is, a single dimension (presumably) knowledge explains all performance on the vignettes. While clearly not true across countries, within countries this is validated both by looking at the drop-off of eigenvalues from a factor model and by specifically testing whether other attributes of the doctor remain significant after including competence as an explanatory variable for performance on the vignettes (Das and Hammer 2005 and Leonard and Masuti 2006).

12
The competence index derived from Item Response Analysis can be compared with characteristics of the provider such as experience, training and institutional affiliation and characteristics of the area that the provider practices. These correlations are explored in Das and Hammer (2007), Leonard and Masatu (2007) and Barber and Gertler (2007), and provide a number of interesting insights.

Take for instance, the perennial question of whether doctors in the public or private sector are better. A typical assumption is that the private sector provides higher quality at a higher price. Yet, this is clearly not true for measures of competence—in all three countries, the public sector is on average as good as the private sector. This seeming equivalence arises from a number of different reasons. In Indonesia, many private sector doctors are those employed by the public sector, and also work in private clinics—so it is not surprising that their levels of competence match up. In India, within the public sector there is a distinct difference between doctors staffing prestigious hospitals and those in primary health clinics. In the private sector, fully trained doctors are slightly better than those in the public hospital (but not by much) and, indeed are likely to be the same people at different stages in their careers. In contrast, the variety of less-than-fully trained practitioners in the private sector is, on average, less knowledgeable than their public counterparts in primary health clinics (but not by as much as you’d think). The overall result is that the public and private sectors look very similar in their competence; at the same time, the distribution shows a long tail of poorly trained doctors in India’s private sector who, by law, cannot practice in public clinics.

Or consider the relationship between training, experience and competence. As expected, training is highly correlated with competence. However, in these data there is no relationship between experience and competence, suggesting either that “learning on the job” is not important 13
or that the better training received by a more recent cohort offsets the advantages of experience for older cohorts.

The relationship between poverty in the area that the doctor practices in and his or her competence stands out. In India, Indonesia, and Tanzania, doctors in poorer areas are less competent than those in richer areas. While this is expected from private sector location decisions since willingness to pay plausibly increases with income, part of the rationale for public sector provision was to enhance equity by offsetting the adverse effects of private sector location decisions. Clearly, this is not happening. Rich people get reasonable care either from public clinics or from highly qualified private physicians. Poor people get care from providers with very deficient competence wherever they go – public clinics or private.

This implies that “access to care” can be very different from “access to quality care”, and more so for the poor. In the Arusha region of Tanzania for instance, most of the rural population lives within 5 kilometers of a functioning health facility. However, many of these facilities are staffed by doctors with such low levels competence that patients are likely to be misdiagnosed and receive the wrong treatment. Low-competence doctors are more common in the rural areas; consequently an average rural resident has to travel further to reach a competent provider than an urban one. Thus, the picture of access derived from distance to the nearest health facility is very different from that derived from distance to the nearest competent provider (Klemick, Leonard, and Masatu, 2007).

Vignettes offer intriguing insights into the training, orientation and knowledge of medical providers in low-income countries. However, what providers say they would do in a hypothetical or role-playing situation and what they actually do when confronted with a patient can be very different, and we turn to this next.

14
# **What Do We Learn From Watching Doctors?**

Direct observations of health care providers have been made in three countries: India, Paraguay and Tanzania. We first describe the differences in method across the countries and then draw some lessons from the three studies, including lessons where we draw linkages between what doctors recommend on the vignettes and what they actually do in practice.

# _How Observation Was Done_

Slightly different methods were used for direct observation of clinical practice. In India and Paraguay, interviewers with no medical training sat for a day with each of the doctors for which vignettes were given. The interviewer recorded details of every interaction, including the time spent, the questions asked, the examinations performed, the treatment dispensed and the price charged. The time spent, questions asked, and examinations performed were then aggregated into a single “effort-index” using principal components analysis. In addition, for two common ailments (diarrhea and cough without fever) that had also been included in the vignettes, the interviewers noted whether the provider asked specific questions that were deemed essential by the team of experts. This allowed a direct comparison between what they knew (and what we knew they knew) and what they did.

In Paraguay, direct observation was supplemented with exit surveys of patients as they left the clinic. Interviewers conducting the exit surveys completed a short module on the patient’s socio-demographic details, their health status (measured through self-reported health status and activities of daily living), their satisfaction with the interaction and a measure of their wealth using the possession of durable goods. These patient details were then matched to the 15
details from the interaction to see how responsive public sector doctors are to patient characteristics.

In Tanzania, where all doctors studied largely followed the same science-based approach to diagnosis, the researcher (who was also a doctor) filled a checklist of expected items, including history taking questions and physical examination procedures. The research team in Tanzania designed instruments to compare the effort provided in consultation for three common conditions to nationally designed protocols for those conditions. Thus, if a patient presented with a cough, fever or diarrhea, there are a series of history taking and physical examination procedures that are clearly rational and useful; doctors who provided more of these items were deemed to provide higher quality care. In addition, doctors who were observed consulting patients with any of these three conditions could be compared to other doctors in the sample, even if their case-mix was otherwise very different. Thus, the percentage of items provided from among those required by protocol was used as a process quality measure and compared across doctors within the sample to examine the determinants of variation in process quality.

One concern with measuring doctor effort through direct observation is that the doctor may work harder in the presence of the research team. The “Hawthorne effect,” as the effect of observation on performance is called, is well documented in health care studies.<sup>3</sup> Leonard and Masatu (2006) examine data on process quality using exit surveys that ask specific questions about the doctors’ activities just before the research team arrived and compare it to data on process quality when the doctor realizes that he is being observed. When the research team arrives at a facility, process quality increases by 20 percent on average. However, for the average

> 3  The Hawthorne effect is named after a study of the Hawthorne Works, an industrial plant located outside Chicago, from 1924 to 1932. The study was originally intended to determine the optimal lighting for the plant. However, the researchers found that workers increased their effort when lighting went either up or down – as long as someone was paying attention to them.

16
doctor, this increase in quality is temporary. Within 10 to 15 consultations (1 to 11⁄2 hours), the doctor has returned to levels of quality exhibited before the research team arrives. Although the reasons for this rise and subsequent fall in quality are not fully understood, it seems likely that doctors put in more effort when they know they are being watched, especially if the observers are their own peers (recall that doctors were used as observers in Tanzania). Despite this reaction to being studied, it appears that process quality measures can be reliably inferred if the research team is willing to wait long enough for quality to return to normal levels. To the extent that Hawthorne effects also bias the measurements of effort upwards, the “true” quality of care is worse than that documented here.

# _Variation across Countries_

Table 2 shows the average time that a doctor spends with every patient, the number of questions asked, and the number of examination procedures actually used. There is substantial variation in clinical style between countries. Far and away the most time and effort appear to be given by practitioners in Paraguay. The average time spent with patients in Paraguay is similar to that in many high-income rich countries. At over eight minutes per visit, Paraguay ranks higher than Germany and Spain, though lower than the United Kingdom or the Belgium.

In contrast, consider Delhi. In the average interaction, the doctor sees the patient for 3.8 minutes, asks 3.2 questions and performs less than one examination procedure. This is low by any standard. In Tanzania, doctors exert more effort in consultations--approximately 6.3 minutes on average. Looking across a wider range of countries in the bottom panel of the table, while the time spent with the patient varies dramatically across countries, this variation is only loosely associated with the country’s per capita national income.

17
# _Variation across Doctors_

Large variations of time and effort given to patients in different countries can be decomposed into that part which varies between doctors—say, due to general differences in the style of practice common to a particular practitioner—and that part that is attributable to variations in time spent by any particular doctor. For India and Paraguay, the variance can be divided about 50/50 between inter-provider and intra-provider variation. In Tanzania, there is also substantial variation in the clinical habits between doctors. Twenty-eight percent of doctors spend three minutes, ask two questions and perform no physical examination with the average patient. This result is frightening, particularly since in all cases where we observed doctors, they were the first point of contact with the medical system for the patient—the numbers from UK and Spain for instance, represent the time spent with the doctor **after** a nurse or assistant has taken basic health measurements for the patient.

We do not have nearly enough information to explain variation between doctors. Only about 40 percent of the variation in India, for example, is explicable with observable characteristics. For instance, in both India and Tanzania, the doctor’s age, gender and experience are not related to effort. The tenure of the doctor in his or her current position is inversely related to the amount of time devoted to each patient. This may be a problem, but it may also reflect greater familiarity with the patient base.

What _do_ jump out of the data are the systematic differences in effort due to institutional affiliations. Variations in India are instructive. In the private sector and in public hospitals, the amount of time spent is fairly low relative to other countries. In the public clinics, however, the situation is disastrous. In these clinics, representing about one third of the visits that we 18
observed, the average number of questions asked was one (and that one often asked rudely). The average time spent was less than two minutes and the average number of examinations was .8, or less than one examination for every patient seen. That is, if a person says they have a fever, in most cases the doctor will not feel or otherwise take the temperature of the patient.

In Tanzania, the split between those who work for the public sector and those who work for nongovernmental organizations makes most of the difference. The latter put in much more effort than the former and effort does not vary much with the rates of remuneration. In Paraguay we do not observe private sector doctors, but do find a clear difference in effort across doctors with permanent or temporary contracts.

In all three countries, incentives may be driving these patterns. In India public primary health physicians are paid by salary and are not monitored. India’s public primary health physicians are usually of much higher social and economic status than are their patients, which influences their attitudes towards the patients and contributes to a very small likelihood that complaints about their behavior will be reported at all or acted upon if reported. As a result absenteeism is high; care is lackadaisical and often rude. In India, public hospital-based physicians, on the other hand, put in much greater effort per patient relative to public primary health care physicians—in fact, the best of them look a whole lot like their private counterparts. We speculate that such workers are better monitored. Also, while it is still virtually impossible to fire a public doctor, the urban hospitals that we visited are coveted jobs and this particular set of doctors may be on a career track that makes them sensitive to the evaluations of their senior supervisors Using data from Tanzania, Leonard et al. (2007) develop an index of incentives that measures the degree to which authority over decision-making is decentralized. Facilities that 19
have a high index value are those in which the chief of post has the authority to hire and fire, choose the qualifications of his staff, set prices, and decide how funds will be used. In contrast, a facility with a low index value is one in which all of these decisions are made in the capital without the input of the staff. As incentives increase, so does effort among doctors in the facility. The story in Paraguay is a bit more complicated. In general, men do less work than do women, and the higher-paid practitioners work less. However, there is a distinct split between how the genders react to incentives. Doctors can hold either permanent contracts or temporary contracts, and in some cases, hold both types of contracts in different institutions. A sample of doctors who held both types of contracts were observed, and the results shows that men put in more effort in their permanent positions and women in their temporary positions. One explanation for this gender difference is that salaries in the Paraguayan health system are negotiated, with strong evidence of discrimination against women. Most male doctors expected their temporary contracts to be renewed, irrespective of job performance, and their increases in salaries to come from negotiations based on their permanent positions. Women felt that salary increases were hard to negotiate and also feared losing their temporary jobs if they did not perform well.

Although these patterns can be plausibly tied to the structure of incentives within the public system, there are several alternate hypotheses. First, public doctors may see many more patients. Second, the presence of observers may have differential effects on public and private providers. This shows up either in reluctance to accept “under-the-table” payments in the public sector or through differential Hawthorne effects for public and private sector doctors. Third, some have suggested that doctors, being underpaid, work about as long as their salary would 20
justify. Fourth, doctors may select into public or private practices depending on their intrinsic motivation and/or cost of effort.

As to workload, the results in India continue to hold after controlling for patient load. While the doctor’s heavier patient load in the public sector is negatively correlated with effort, the difference between the public and private sector in effort remains large and significant with additional controls for the average number of patients the doctor sees in his or her practice.

Further, the cases seen in both public and private sectors are identical. If anything, judging by the extra number of days people wait to go to public clinic, the severity of their disease is likely to be greater than in private and should warrant more time and attention rather than less. Similarly, in Tanzania, the effort provided by rural doctors, who also have lower caseloads is much lower than that provided by urban doctors with higher caseloads.

While there _are_ differences in the Hawthorne effect for public and private providers, but these accentuate the magnitude of the effort-gap between the two sectors. Leonard and Masatu (2007b) show that this effect is strongest for doctors who normally face the lowest incentives to provide quality.

We do not know anything about bribery, since we did not observe any and this could have been induced by our observations. However, if under-the-table payments are important, it would suggest that doctors _are_ responding to price incentives—when you pay them more for a specific service, they work more.

Do public doctors put in less effort because they are paid to little? While complicated, there are some indications in each of the countries that raise doubt about the extent of this problem. In India, income can be estimated controlling for quality of the practitioner on the basis of vignettes and it turns out that public PHC doctors are not likely to make more in the private 21
sector. We would predict they would make much less. In Tanzania, effort is unrelated to remuneration. And in Paraguay, doctors who are paid more work less.

Less effort in the public sector is almost surely partly due to selection on the cost of effort. This underscores the impact of incentives, not only on what doctors do in their chosen profession, but also on how they choose between the two sectors to begin with. If you expect to work less you may very well choose to stay in the public sector. We cannot disentangle these two effects with the existing data.

# _Explaining Variation for Individual Doctors_

In Paraguay, there is almost no correlation between variation in doctor effort and the characteristics of the patients that doctor sees (Das and Sohnesen, 2007). On the positive side, this finding implies that doctors in Paraguay do not discriminate by the background of the patient; on the negative side, it implies that doctors did not expend greater effort or spend more time on those who were sicker. In India, private doctors expend less effort on their younger patients and on their repeat patients; in the public sector, the differences are statistically and qualitatively smaller. Overall, only about one third of the variation observed within doctors is explicable with easily observed characteristics of the patients.

# **What Do We Learn From Both Testing and Watching Doctors?**

Testing and watching doctors suggests that the variation in quality of medical advice arises from (a) variation in competence, defined as what doctors know; (b) variation in effort, defined as how hard doctors work; and (c) implicitly, the link between the two. The connection between competence and effort could go either way. For example, greater competence could 22
mean less time spent in making a correct diagnosis and offering a correct treatment. Conversely, a knowledgeable doctor may take more time in being more thorough.

The data suggest two patterns. First, greater competence is associated with higher effort. For instance, in India we divided doctors into four categories: with and without an MBBS, and private versus public hospital-based (or clinic-based) doctors. In all four categories, the more competent the provider, the more effort exerted. Indeed, the increased effort from greater competence looks similar between the groups, except for the deficient effort in public clinics that stands out. The finding that effort and competence are positively correlated is confirmed in the data from Tanzania as well (Leonard and Masatu, 2005).

Second, despite the positive correlation between effort and competence, there is a large and significant gap between what doctors know they should do (as measured by testing doctors) and what they actually do (as measured by watching doctors). In both India and Tanzania, for patients reporting fever, cough or diarrhea, observers were asked to note whether the doctor asked certain questions and performed certain examinations that had been included in the vignettes as essential tasks. Since these cases had also been covered under the vignettes, we can directly compare what doctors said they would do and what they actually did when faced with a similar patient. How large is the gap between knowledge and practice?

For the data from India, we considered the percentage of essential tasks completed in the vignettes concerning diarrhea and “cough without fever” and in direct observation of patients reporting these conditions for three types of doctors—private doctors with an MBBS degree; private doctors without an MBBS degree (recall that doctors with an MBBS degree are more competent) and; public doctors, all of whom have an MBBS degree. There are several noteworthy features of the comparison. First, private doctors without an MBBS complete just 23
over 20 percent of all essential tasks, but they are doing pretty much all they know to do—the constraint on their performance is competence. Second, private doctors with an MBBS knew 40 percent of the essential tasks, but in actual practice were completing only 25 percent of them. The constraint on their performance is effort. Third, the gap between competence and practice among public sector doctors is an order of magnitude higher—these doctors knew to complete 30 percent of essential tasks, but actually completed only 8 percent. Here, the constraint on performance is clearly effort.

In Tanzania, similar analysis tells a slightly different story. For items on the direct observation checklist that also appear on the vignette, doctors in the public and nongovernment organization sector both completed 50 percent of all essential tasks. However, whereas doctors in the nongovernment organization sector completed 44 percent of these tasks when observed with their regular patients, doctors in the public sector completed only 36 percent of the same tasks. The items that correspond across the vignette and the direct observation checklist are not the same for Tanzania as they are for India, and therefore the higher percentage of task completed should not be interpreted as higher quality, as Leonard and Masatu (2005) discuss. However, the numbers illustrate the fact that there are no differences in competence between the public and nongovernment organization sectors in Tanzania, but that there are significant differences in effort.

Two implications of these results seem worth emphasizing. First, the poor in these countries receive poor quality medical advice both because the medical providers located close to them are less competent and because less competent doctors exert less effort. One of the ostensible rationales for a public sector presence in health was precisely to ameliorate the detrimental effects of profit-maximizing behavior among private sector providers. Yet the 24
association between competence and local poverty is identical for doctors in the public and private sector. Worse, effort in the public sector is so low that it is often better to go to an untrained provider in the private sector than a trained doctor in the public sector because the greater effort makes up for the lower level of competence.

In low-income African countries one option may be the nongovernment organization (NGO) sector. The NGO sector in Tanzania, as in many other African countries, is an institution handed down from earlier missionary efforts and plays a significant role in education and health. In Tanzania, the NGO sector is staffed with Tanzanians who graduate from the same medical schools as doctors in the public sector (as reflected in their similar levels of competence), but the data suggest that some features of their organizational structure encourages these doctors to exert more effort (Leonard, 2002). NGOs are more expensive than their public counterparts, but they are located in poor areas and Leonard and Masatu (2007) suggest that they may do a better job of serving the rural population of Tanzania.

A second implication is that the link between knowledge and practice provides stark evidence that “training doctors”—a perennial favorite among multilateral institutions and donors—is just not going to help. The effect of training on the quality of medical care can be divided into two parts: the effect of training on the knowledge (as measured by the probability of completing essential tasks in vignettes) and how knowledge affects effort (as measured by the probability of completing essential tasks in practice). Both these effects are small.

In Tanzania, for example, doctors are much less likely to get the correct diagnosis in practice than their competence would suggest. In these data, a one standard deviation change in competence corresponds approximately to moving from a clinical assistant (with four years of secondary schooling and two years of medical training) to a medical officer (with six years of 25
secondary schooling and five years of university level medical training). These five years of additional training on average correspond to a 5 percentage point increase in the probability that the doctor would correctly diagnose the illness. When measured by actual behavior, however, the difference between a clinical officer and a medical officer for practice quality is about 1 percentage point. Based on these estimates, two additional years of school and three additional years of medical school buys an increase of only 1 percentage point in the quality of medical care actually delivered. Moreover, training doctors in _poor_ areas also raises the question of where they will locate after they are trained. If the better-trained doctors follow the typical location patterns, it is unlikely that they will remain in the poorer areas.

# **Research Agenda and Concluding Thoughts**

Some of the patterns that we have found in India, Indonesia, Paraguay, and Tanzania are strong enough that we feel comfortable presenting them as likely to be universal. In particular, the competence of doctors in low-income countries is low, the quality of care provided to patients is even lower than would be suggested by a doctor’s competence and the poor have access to worse quality care than the rich, whether from the public or the private sector. However, these facts and their interrelationships require further exploration. Several different research agendas are suggested by these results.

First, research is required to establish the link between the quality of medical advice measured using these tools and health outcomes. A few studies suggest that both competence and effort are positively correlated with health outcomes, but clearly further work is required. For example, using a decline in health care provider quality as measured by vignettes in Indonesia between 1993 and 1997 (caused by a funding freeze), Barber and Gertler (2005) demonstrate a 26
link between competence and children’s heights and weights. Similarly, Barber and Gertler (2007) show in Mexico that the birth-weight of children increased in villages with conditional cash transfer programs, whereby mothers were given cash if they completed a certain number of doctor visits. What is of particular interest in their study is that the improvement in birth-weight had little do with greater doctor visits—in fact, these barely changed as a consequence of the program. Using data on the percentage of essential tasks completed in maternal check-ups, the authors argue that community meetings “empowered” women to demand better care from their doctors, and that, consequently, there was a large increase in effort and hence the quality of care delivered. Similar results are documented by Bjorkman and others (2006). In a randomized study on the effect of citizen report cards for health care outcomes, the authors document an improvement in child health, which they attribute to greater effort among providers.

A second issue is how to deal with sorting effects, which can arise in a variety of ways. For example, our measure of effort is based on observations of patients who choose to visit the doctor observed. But if those with high incomes visit more competent providers, more competent doctors may provide more effort because they are being paid more, and not because they are more competent. Das and Hammer (2007) use the presence of multiple doctor clinics with random sorting between doctors to show that the correlation between competence and effort was identical in the entire sample and in this more specialized setting. In Leonard (forthcoming), the Hawthorne effect is used as an instrument for quality; patients who receive their consultations immediately after the research team arrived receive better consultations for reasons that are uncorrelated with patient characteristics. This method does not expose the determinants of case mix, but allows the researcher to link effort to outcomes (patient satisfaction, in this case) despite case mix variation. Another possible way to control for case mix is to collect more patient data 27
through detailed exit surveys, so that it becomes possible to adjust for a wide array of patient characteristics. Another approach, which has been used to study the quality of health care in high-income countries but not in low-income countries, is to use fake patients with standardized symptoms, which is another way of holding patient characteristics constant. The ethical, feasibility and estimation issues with either approach are yet to be worked out.

A third issue is the structure of market equilibrium, recognizing that location choices of doctors are endogenous. On average, less competent providers are in poorer areas, but considerable variation exists. Why do we find competent providers in poor localities when, a short distance away, their competitors are making more money? What is the process in the public sector whereby less competent providers are assigned to poorer areas? Some of these answers must be related to the demand for health care as a function of quality. Do high-income people see better doctors because they are more educated consumers and better judges of quality, or do better doctors choose to locate where markets are more lucrative? In recent work, households were asked to rank the same set of doctors that vignettes were administered to in India. Preliminary results suggest that the overall correlation between household’s ranks and vignettes performance is fairly high, although there is substantial variation across localities (rich people do worse). Furthermore, prices reflect competence, although the relationship disappears once observable characteristics of the doctors (his/her qualification and the neighborhood the practice is located in) are included (Das, Do and Hammer, 2007). These results are a start, but further work is required to understand the market equilibrium that simultaneously explains location and pricing as a function of competence.

In concluding, we submit that the research agenda on quality of medical care in lowincome countries has contributed to our understanding of health care in three ways. First, it has 28
highlighted that standard measures of health care quality in low-income countries, based on physical infrastructure and sometimes on availability of certain drugs, are sorely inadequate. The quality of medical advice cannot be “proxied for” by measures of physical infrastructure. Second, it has shown that measuring the quality of medical advice using vignettes and direct observation _is_ feasible, if somewhat more difficult. Yet, this is clearly the relevant dimension along which variations—within and across countries—matter. We have shown the quality of care to be extremely low and been able to show how this arises from a combination of low competence and low effort. Third, the agenda has also highlighted issues that arise once the quality of medical advice is accurately measured. The question of how market equilibrium is arrived at in settings with public and private providers of differing quality is of particular interest; further research along these dimensions could help understand the structure of market failures in the delivery of health care in low-income countries.

29
# **Acknowledgements**

We thank Esther Duflo and Paul Gertler for their support and helpful comments at various stages of this work. We are also grateful to the editor of the _Journal of Economic_ _Perspectives_ , Timothy Taylor and reviewers James Hines and Jeremy Stein, whose comments have added greatly to the paper. Das and Hammer acknowledge funding from a World Bank research grant. The findings, interpretations and conclusions expressed in this paper are those of the authors and do not necessarily represent the views of the World Bank, its Executive Directors, or the governments they represent.

30