Neonatal Asphyxia A mother gives birth. The newborn is not crying. The newborn fails to establish regular breathing and appears pale and slightly blue.

There are a number of ways of scoring performance in patient case simulations, or vignettes. In this study we focus on diagnostic and treatment accuracy. We identified the lowest common denominators across the country-specific protocols to define comparable criteria across countries. This methodological choice implies that knowledge is assessed at a lower standard than more stringent WHO protocols.

The medical vignettes are used to measure clinic knowledge. That is, a health worker’s clinical vignette performance is not a measure of quality of care per se. It does, however, represent a good measure of the quality care which the provider can deliver, assuming he/she has access to relevant medical infrastructure. There are alternative methods to test for clinical knowledge and performance, including direct observation of real patient consultations and the use of standardized patients. Observation of real patient consultations, like vignettes, may be subject to observer effects (also referred to as Hawthorne effects). A study in the U.S. finds that the quality score derived using data from vignettes was similar to the quality score with standardized patients (i.e., trained actors who make unannounced visits to the clinic), but a study in India shows large gaps between vignettes and actual practice with standardized patients.<sup>7,8</sup> A study using data collected from 104 medical providers in northern Tanzania find that item by item, vignettes and direct observation produced identical results 63% of the time.<sup>8</sup> Aggregate scores derived from vignettes tend to be higher than scores from direct clinical observation, suggesting that knowledge measured using vignettes should be interpreted as an upper-bound of what providers could do; i.e., the vignettes measure what the medical provider knows, but not necessarily whether the provider applies this knowledge in practice.<sup>9,10</sup> The main advantages of using medical vignettes compared to direct observation of real patient consultations (or re-examination of patients outside of the clinic by well-qualified personnel), are that they are far less time-consuming and they increase comparability by avoiding issues related to differences in case mix. Moreover they provide a direct measure of providers’ clinical knowledge, rather than a combined measure of knowledge and effort. Data from medical vignettes is also directly policy relevant as the scores are closely linked to the quality of medical training.

Validation of vignettes typically involves comparing scores from vignettes with scores from direct observations.<sup>9,10</sup> This is less relevant for this study as we focus on provider knowledge, realizing that in practice (some) providers may not always fully utilize their knowledge when diagnosing and treating patients. Still, the vignettes utilized in this study has been (indirectly) validated as a testing method for provider knowledge. One source of validation is the fact that providers with a high share of relevant items performed are much more likely to suggest the correct diagnosis and treatment.<sup>9</sup> That is, better knowledge of which history taking question to ask and examinations to perform is highly correlated with both diagnostic and treatment accuracy. The two figures below, repeat that exercise using the full sample of vignettes and share of items performed (89,824 observations). We estimate a logit model with the binary response variable correct diagnosis [correct diagnosis and treatment] and with share of items performed as a continuous predictor and country and condition categorical predictors as control variables. The figures plot the adjusted probability of reaching the right diagnosis [right diagnosis and treatment]. As the share of items performed increases, the estimated probability of reaching a correct diagnosis and correct diagnosis followed by a correctly described treatment increase. The adjusted probability of correct diagnosis and treatment is estimated to be 33% for a 48