Provided that key identifying assumptions are met,<sup>15</sup> the size of the estimated demand-robust confidence interval on the vaccination rate specifies the range of values a survey-based estimate of vaccine take-up that is free from experimenter demand would fall into.

Finally, we can compare the point estimate of vaccine uptake under the standard questionnaire design (T1), along with its demand-robust confidence interval, to the vaccination rate as reported in administrative data sources. The larger the demand-robust confidence interval on our surveybased estimate of vaccine uptake, the larger the potential importance of enumerator demand in explaining misalignment between both data sources. Statistically significant differences between the administrative data and our survey-based estimate, with demand-robust confidence intervals, would suggest that enumerator demand effects alone cannot account for the difference between our survey-based estimates and the administrative data.

# 4 Results

## **4.1 Estimated vaccine coverage in survey and administrative data**

We find substantial misalignment between survey-based estimates and administrative records of COVID-19 vaccine coverage. Comparing phone survey estimates to administrative coverage figures from the same time<sup>16</sup> in a sample of 36 LMICs, we find survey estimates that suggest coverage rates that are between 11.5 percentage points lower and 37.1 percentage points higher than those reported in administrative sources (Figure 1). Vaccination rates are statistically different in 28 out of 36 countries (46 out of 57 survey waves). In 21 out of these 28 countries (35 survey waves), survey estimates suggest higher vaccine coverage than is reported in administrative data (in 7 countries survey estimates are lower). On average, survey estimates exceed the administrative data by 47% when excluding extreme positive outliers occurring below 5% reported administrative coverage. However, this pattern displays noticeable regional variation: While estimated survey rates are fairly aligned with administrative figures in Latin America and the Caribbean (LAC), they systematically exceed administrative reports by large margins in Sub-Saharan Africa.

> 15 Most notably, monotonicity implies that negative reinforcement should not _increase_ the propensity of respondents to report being vaccinated and positive reinforcement should not _decrease_ their propensity. Furthermore, bounding assumes that the “true” survey-based estimate of the vaccination rate, i.e., the estimate absent any demand/social desirability effects, lies within the bounds provided by the estimated vaccination rate under negative and positive reinforcement. This also has implications for how to choose the framing: Framing should be chosen such that the induced demand effect is _just_ stronger (positively as well as negatively) than what is hypothesized to be the case in the original questionnaire design. Inducing a much stronger demand effect leads to demand-robust confidence intervals that are needlessly conservative. In other words: Conditional on inducing a demand effect that is stronger than in the original questionnaire, choosing the question framing that evokes the smallest demand effect is optimal as it leads to the narrowest demand-robust confidence interval within which we expect an experimenter-demand-free survey estimate would lie.

> 16 We use the end of the phone survey data collection period as the time of comparison, which allows for some time lag in administrative data reporting.

16