We find that the discrepancies introduced by denominator issues are non-negligible and amount to a difference of up to 5 percentage points between coverage rates reported in the WHO dashboard and the OWID dataset. On average, reported coverage in the WHO dashboard is 1.6 percentage points higher than that reported in the OWID data. Similarly, coverage reported in the WHO dashboard is on average 1.9 percentage points higher in our sample of five Sub-Saharan African countries. This implies that, on average, the coverage rates reported in the WHO data assume a smaller population size. However, cases also exist in which assumed population size in the WHO data is higher, leading to up to 2.9 percentage points lower vaccination rates.

These discrepancies seem to arise because of differences in baseline years for which population figures are taken. The OWID dataset uses the latest, 2022 UN population projections (Mathieu et al. 2021), whereas population figures implied by the WHO’s reported coverage rates typically correspond to UN World Population Prospects for 2019 or 2020.

# 5 Discussion

Phone surveys have become commonplace during the COVID-19 pandemic and have informed a large body of (health) policy research. Our results from 57 survey waves on COVID-19 vaccination across 36 LMICs indicate that phone surveys may produce estimates of COVID-19 vaccine coverage that differ from administrative figures. In some cases, survey estimates exceed administrative figures by a margin that would suggest vastly different policy conclusions. Such misalignment is particularly concentrated in Sub-Saharan Africa while coverage rates estimated in other regions on average track administrative records more closely.

Upon investigating potential sources of bias, our findings largely maintain the reliability of the phone survey data. We find phone survey estimates from five Sub-Saharan African countries to be robust to a number of commonly feared representation and measurement errors, but they are affected by how respondents are selected (Figure 3 and Appendix Table A3).

We find little evidence that errors of measurement bias the phone survey estimates. Interviewing respondents in person rather than on the phone does not lead to survey estimates that are aligned with administrative records. Similarly, reported vaccine uptake does not differ significantly between first-time and previous respondents of the phone surveys suggesting that panel conditioning effects do not bias estimates. Contrary to Wolter et al. (2022) who use a list experiment in an online sample in Germany to find experimenter demand biasing reported vaccine uptake, we do not find evidence of such effects in our sample. While we find proxy reports to have generally good accuracy, we do find that they miss out on some vaccinations obtained by other household members.

As for sample selection, we find no statistically significant difference in estimated vaccination rates between the phone survey sample of households and the full nationally representative sample of households. However, respondent selection matters. In line with the results of Bradley et al. (2021) from online surveys in the United States, estimated vaccine uptake is significantly lower when taking into account respondent selection effects at the individual level (Figure 3). Moving 28