# **4.4. Limitations**

This study has limitations that are important to consider when drawing conclusions from our evidence. Due to its rapid introduction and promotion on a large scale, our findings about COVID-19 vaccine hesitancy might not be applicable to other long-established vaccines (e.g., MMR, DTaP, flu) or new vaccines (HPV). Recruitment and survey administration on Facebook, while innovative for mirroring national demographics using Facebook targeting and quota sampling, cannot provide a representative sample, leading to under-representation of older age groups, the poorest, and least educated, who might be disproportionally impacted by the COVID-19 and future pandemics. Self-selection bias may introduce systematic differences between respondents and non-respondents based on unmeasured factors and affect the findings. As such, the generalizability of the findings from our meta-analysis to offline populations remains to be tested and the impacts of our studies likely represent upper-bound estimates of effects. Despite these limitations, the sampling strategy of stratifying by demographic characteristics allowed for greater representation and timely data collection among key subgroups (e.g., women) who can be harder to reach in online settings.

The study measured self-reported vaccination intentions, not actual behavior. While behavioral intentions are commonly used to predict behaviors in experiments, and one meta-analysis found that a medium-to-large change in behavioral intention ( _d_ = 0.66) led to a small-to-medium change in behaviors ( _d_ = 0.36) (Webb & Sheeran, 2006), the extent to which changes in intentions translate into increased vaccinations across these international contexts remains unknown and warrants further investigation, when feasible (e.g., Alhajji et al., 2023). Indeed, some research has identified differential impacts of messaging interventions on vaccine intentions compared to behavior (Saccardo et al., 2024).

Lastly, we were limited in exploring additional study-level moderators (e.g., World Value Surveys [2024]; Hofstede’s cultural dimensions [Culture Factor Group, 2024]) given scarcity of data available for LMICs. While we were able to draw from the Wellcome Global Monitor Indicators to explore the moderating role of country-level beliefs and trust in healthcare workers and beliefs about vaccines, these concepts may be better explored at the individual level, as evidenced by modest empirical evidence for aggregation to the country level described in Appendix D. Our analysis focused on only demographic characteristics as individual-level moderators, as other potentially relevant factors were omitted from surveys to keep them brief for participants. Future research could explore additional moderating factors that might have more explanatory power.

# **4.5. Conclusions**

This study is part of the World Bank's COVID-19 response, aimed at diagnosing vaccine hesitancy and related factors, and informing and guiding real-time risk communication efforts across multiple governments simultaneously. While the WHO declared the end of the COVID-19 pandemic in May 2023 (WHO, 2023), ensuring high immunization rates remains crucial for public health. Our findings demonstrate that behaviorally-informed messages have the potential to nudge vaccine-hesitant individuals towards vaccination. Impacts of such messages can represent significant public health gain, as up to 6 million people were vaccine hesitant in these 23 countries and territories by the time of the study. However, one lesson from our study is that a "one-size-fits-all" approach will not work. Our research emphasizes the importance of testing and tailoring messages to specific audiences and contexts, as the effectiveness of some messages varied across countries. This study highlights the value 27