Table 4: Summary of methods to assess provider knowledge and delivery of care. Method Strengths Weaknesses Reliability Feasibility Relative Costs Provider knowledge Provider interviews – knowledge, attitudes Measures practical knowledge; measures provider perceptions Hawthorne effect – Altered behavior or performance of the health worker resulting from awareness of being part of an experimental study; Does not measure actual practice Good for provider perceptions and knowledge (but many poor questions are used) Can be done in large sample sizes, wide variety of cases, including rare events Low Provider vignettes Measures practical knowledge, decision-making Hawthorne effect – Altered behavior or performance of the health worker resulting from awareness of being part of an experimental study; Does not measure actual practice Variability based on interviewer; can have low reliability (though may be good in certain circumstances) Depends on high quality interviewer, uses lower sample size and limited variation in cases, some rare events Higher interviewer qualifications and training required Health service delivery Observation of care Measures actual quality delivered Hawthorne effect – Altered behavior or performance of the health worker resulting from awareness of being part of an experimental study; Measures optimum care delivered Good -- Limited inter-rater problems, but not well documented Requires high case load &/or common conditions; more intrusive than other methods Modest training (e.g. 1 week) of literate or student health workers; supervision costs Exit Interview – History taking and physical exam Lower Hawthorne effect than observation if provider can be blinded to study Does not measure actual sequence of physical exam Poor patient/caretaker recall: low correlations with actual history documented Can do large numbers, but better for common conditions Modest training (e.g. 1 week) of literate or student health workers; supervision costs
Exit interview – Counseling, Perceptions, Patient characteristics Measures what caretaker actually understands; Measures immediate perceptions ; Can measure key characteristics (e.g. wealth) which can be compared with population parameters Perceptions at point of care may not reflect those measured later Good – limited inter-rater problems; some questions have shown good correlation with actual history Can do large numbers if facilities have high volumes, better for common conditions Modest training (e.g. 1 week) of literate or student health workers; supervision costs Simulated clients (mystery patients) Lowest Hawthorne effect if able to keep interviewer blinded Poor verisimilitude for children and women in delivery Not possible for pregnant women and sick children (ethical issues) Variability based on actor Low sample size; least variation in conditions to test Higher interviewer qualifications and training can increase costs Patient record review If good record keeping, can reflect care intended to be provided; Better for diagnosis and treatment (if systematically recorded) Poor records are the norm; Limited information on actual tasks performed reflects what providers say patient care was rather than actual care delivered; Poor, especially for record keeping Non-intrusive, quick, but records rarely of adequate quality for use other than volume of service Cheapest