# **2. Method**

## **2.1. Study Sample**

To identify whether different behaviorally informed vaccine messages can increase intent to vaccinate, we conducted 28 randomized survey experiments using social media in 23 countries and territories between January 2021 (when vaccines were not widely available) and June 2022 (when vaccine supply was widely available). See Appendix A for a list of sampled countries and when studies were implemented in each country. Multiple studies were conducted in Lebanon, Iraq, and Jordan throughout the course of the pandemic. Each study followed similar protocols for recruitment, experimental design, and survey measurement, which are described below. The results from individual studies have been reported in other publications for Belize, Haiti, Jamaica (Margolies et al., 2023), Papua New Guinea (Hoy et al., 2023), Saudi Arabia (Lim et al., 2024), and Zambia (Hoy et al., 2022). Also, a separate survey with family doctors in Romania (Badiani-Magnusson et al., 2022) was conducted as part of the same global research program, yet was not included in this IPD meta-analysis. Our experiments were approved by the Health Media Lab Institutional Review Board (HML IRB), approval number # 1017TWBG21.

## **2.2. Participant Recruitment and Sampling**

Participants were recruited through Facebook using quota sampling by displaying ads inviting participants to interact with a chatbot in Facebook Messenger to complete the survey (using languages of preference) for a chance to win a lottery reward.

The sample in each country was drawn from all Facebook accounts 18 years and older in the country. We used quota sampling by exposing the recruitment ads to pre-defined demographic groups based on age, gender, and region to mirror the country’s populations. To get a quota sample and optimize recruitment, we updated the ad placement in real-time based on participating Facebook users’ characteristics to ensure proper representation of each demographic group in the final sample using marketing applications such as the Virtual Lab (Rao et al., 2020) and RevealBot (RevealBot, 2024). For example, if a subgroup was underrepresented and had higher attrition or non-response, then the budget for ads targeting that demographic subgroup was automatically increased to better approximate the required sample for that subgroup.<sup>4</sup> Across all studies, the final sample included 229,886 responses in 28 studies with an average of 7,800 responses per online study. Only participants who had not received the COVID-19 vaccine were included in the experiment ( _n_ = 123,720), which is the focus of the results reported in this paper. No

> 4 For example, we recruited participants in Iraq targeting ads based on a combination of gender, age group, and governorate. This combination created x strata for recruitment with each stratum reflecting a combination of these three characteristics. For instance, one set of ads targeted women aged 40-49 who lived in the governorate of Baghdad while another set of ads targeted men aged 40-49 who lived in the governorate of Baghdad with quota targeting based on the latest available population data. During data collection for the study, if the proportion of participants from this demographic group in the study sample was lower than its proportion in the broader population, the recruitment ads were dynamically adjusted to spend more money to better target this population and increase its proportion in the study sample.

6