metadata: authors: 0: author: Pinzon Hernandez,Daniel Alejandro
1: author: Lim,Jungkyu Rhys
2: author: Dugas,Michelle Anne Matejovits
3: author: Moscoe,Ellen Elizabeth
4: author: Chatila,Mohamad
5: author: Cameron,Corey Morales
6: author: Vakis,Renos
7: author: Afif,Zeina
8: author: Orozco Olvera,Victor Hugo
count: Papua New Guinea,World
docna: 0: docna: Behaviorally Informed Messages Increase
            COVID-19 Vaccination Intentions : Insights from a Global Meta-Analysis
docty: Policy Research Working Paper
theme: Social Inclusion,Participation and Civic Engagement,Human Development and Gender,Data Development and Capacity Building,Social Development and Protection,Disease Control,Health System Strengthening,Pandemic Response,Health Service Delivery,Public Sector Management,Data production, accessibility and use,Health Systems and Policies
abstracts: cdata!: During the COVID-19 pandemic, low- and
            middle-income countries struggled with lower vaccination
            rates compared to wealthier countries, posing challenges to
            reducing virus transmission, mitigating healthcare system
            pressures, and promoting economic recovery. Communications
            campaigns offer low-cost opportunities to overcome such
            challenges by strengthening vaccine confidence and
            intentions to get vaccinated, but empirical testing is
            needed to identify which messages will be most effective in
            different contexts. To support policy-making efforts to
            design effective communication rapidly during the pandemic,
            a global research program of 28 online experiments was
            conducted by recruiting respondents (123,270 individuals)
            through social media between January 2021 and June 2022
            across 23 mostly low- and middle-income countries and
            territories. An individual participant data (IPD)
            meta-analysis of these data summarizes the results of this
            research program testing the impact of behaviorally informed
            messaging on vaccine intentions. Results from the
            meta-analysis show that among unvaccinated survey
            respondents, behaviorally informed messages significantly
            increased the odds of vaccination intention by 1.28 times
            overall and up to 1.93 times in individual studies (safety
            messages in Papua New Guinea). Significant pooled effects of
            specific framings ranged from increasing the odds of
            vaccination intention by 1.16 times (variant framing) to
            1.45 times (experts and religious leaders framing).  This
            research underscores the importance of communication
            tailored to address different drivers of vaccine hesitancy
            and offers insights for handling future health crises with
            behavioral communication strategies leveraging rapid
            insights afforded by social media.
display_title: Behaviorally Informed Messages
            Increase COVID-19 Vaccination Intentions : Insights from a
            Global Meta-Analysis
listing_relative_url: /research/2024/11/34424990/behaviorally-informed-messages-increase-covid-19-vaccination-intentions-insights-global-meta-analysis
topicv3: Health Education, Information, and
            Communication m1327026 1463,Social Media m1327099 662,Immunizations m1327505 1459,Behavioral Initiatives for Poverty
            Reduction and Equity m1503826 3746,Behavior Change in Public Health m1504351 3656
publishtoextweb_dt: 2024-11-19T00:00:00Z
dois: 10.1596/1813-9450-10981
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
no_of_pages: 57
author_keys: 0: author_key: 1723058
1: author_key: 2117014
2: author_key: 2125751
3: author_key: 1206851
4: author_key: 1808182
5: author_key: 2053506
6: author_key: 167914
7: author_key: 167285
8: author_key: 810348
model_extractions: Policy Research Working Paper 10981

# Behaviorally Informed Messages Increase COVID-19 Vaccination Intentions

Insights from a Global Meta-Analysis _Daniel Alejandro Pinzon Hernandez JungKyu Rhys Lim Michelle Dugas Ellen Moscoe Mohamad Chatila Corey Cameron Renos Vakis Zeina Afif Victor Hugo Orozco Olvera_ Development Economics Development Impact Group November 2024 A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click **here** for direct access.
Policy Research Working Paper 10981

# **Abstract**

During the COVID-19 pandemic, low- and middle-income countries struggled with lower vaccination rates compared to wealthier countries, posing challenges to reducing virus transmission, mitigating healthcare system pressures, and promoting economic recovery. Communications campaigns offer low-cost opportunities to overcome such challenges by strengthening vaccine confidence and intentions to get vaccinated, but empirical testing is needed to identify which messages will be most effective in different contexts. To support policy-making efforts to design effective communication rapidly during the pandemic, a global research program of 28 online experiments was conducted by recruiting respondents (123,270 individuals) through social media between January 2021 and June 2022 across 23 mostly low- and middle-income countries and territories. An individual participant data (IPD) meta-analysis of these data summarizes the results of this research program testing the impact of behaviorally informed messaging on vaccine intentions. Results from the meta-analysis show that among unvaccinated survey respondents, behaviorally informed messages significantly increased the odds of vaccination intention by 1.28 times overall and up to 1.93 times in individual studies (safety messages in Papua New Guinea). Significant pooled effects of specific framings ranged from increasing the odds of vaccination intention by 1.16 times (variant framing) to 1.45 times (experts and religious leaders framing). This research underscores the importance of communication tailored to address different drivers of vaccine hesitancy and offers insights for handling future health crises with behavioral communication strategies leveraging rapid insights afforded by social media.

This paper is a product of the Development Impact Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at dpinzonhernandez@worldbank.org, rhyslim@worldbank.org, and mdugas@worldbank.org. A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click **here** for direct access.

<!-- Start of picture text --> TRANSPARENT<br>ANALYSIS<br>G<br>R E<br>N<br>S<br>I P<br>Y E<br>K A<br>C A<br>R P<br>LI R<br>O E<br>O C<br>W R<br>H<br>S<br>P<br><!-- End of picture text --> _The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Behaviorally Informed Messages Increase COVID-19 Vaccination Intentions: Insights from a Global Meta-Analysis**

Daniel Alejandro Pinzon Hernandez<sup>1,2</sup> <u>dpinzonhernandez@worldbank.org</u> JungKyu Rhys Lim<sup>1,2</sup> <u>rhyslim@worldbank.org</u> Michelle Dugas<sup>1,2</sup> <u>mdugas@worldbank.org</u> Ellen Moscoe<sup>1</sup> <u>emoscoe@worldbank.org</u> Mohamad Chatila<sup>1</sup> <u>mchatila@worldbank.org</u> Corey Cameron<sup>1</sup> <u>ccameron90@gmail.com</u> Renos Vakis<sup>1</sup> <u>rvakis@worldbank.org</u> Zeina Afif<sup>1</sup> <u>zafif@worldbankgroup.org</u> Victor Hugo Orozco Olvera<sup>1</sup> <u>vorozco@worldbank.org</u>

**_JEL Codes:_** I10, I12, I18, D91

**_Keywords:_** infectious disease, coronavirus, immunization, message framing

> 1 World Bank Development Impact (DIME).

> 2 Corresponding authors: Daniel Pinzon (dpinzonhernandez@worldbank.org), Rhys Lim (rhyslim@worldbank.org), Michelle Dugas (mdugas@worldbank.org).
# **Acknowledgments**

The authors express their gratitude to all who contributed to the implementation of this project, especially colleagues from regional and global Health, Nutrition, and Population (HNP) teams, Poverty and Equity teams, External and Corporate Relations (ECR) teams, the Virtual Lab, Meta for their ad credits and support, and country government partners. The authors would like to thank Aneesa Arur (Program Leader, HAEDR), Reena Badiani-Magnusson (Senior Economist, Program Leader, EECDR), Tom Bundervoet (Lead Economist, EAEPV), Agnes Couffinhal (Senior Economist, HHNGE), Laura Di Giorgio (Senior Economist, HLCHN), Olena Doroshenko (Senior Health Economist, HECHN), Yohana Dukhan (Senior Health Economist, HAWH3), Denizhan Duran (Senior Health Economist, HMNHN), Diya Nitham Mousa Elfadel (Health Specialist, HMNHN), Rochelle Se Yun Eng (Senior Health Economist, HEAH1), Takahiro Hasumi (Senior Health Specialist, HECHN), Christopher H. Herbst (Senior Health Specialist, Program Leader, HMNDR), Christopher Alexander Hoy (Economist, EPVGE), Timothy A. Johnston (Manager, IEGHC), Lombe Kasonde (Senior Health Specialist, HHNGE), Phillis Kim (Consultant, HHNGE), Haena Kim (Consultant, EPVGE), Leonardo Ramiro Lucchetti (Senior Economist, EAEPV), Fernando Xavier Montenegro Torres (Senior Health Economist, HAEH1), Son Nam Nguyen (Lead Health Specialist, HMNHN), Nicolas Rosemberg (Senior Economist, Health, HAWH3), Sering Touray (Senior Economist, EAWPV), Opope Oyaka Tshivuila Matala (Senior Health Specialist, HAWH2), Sherin Varkey (Program Leader, HAEDR), David Wilson (Director, HHNDR), and Laura Zoratto (Senior Economist, EGVPI). The team is grateful to Alexandru Cojocaru (Senior Economist, EECPV) and an anonymous peer reviewer for their valuable comments as peer reviewers, and to Charles Arnold for his input into an earlier version of this paper. This study was partially funded by the Advancing Health Online (AHO) Initiative, a fiscally sponsored project of Global Impact and funded by Meta and MSD to advance public understanding of how social media and behavioral sciences can be leveraged to improve the health of communities around the world. The findings, opinions, interpretations, recommendations, and conclusions expressed herein do not reflect the views of the World Bank, its board of executive directors, or the governments they represent.

2
# **1. Introduction and Literature Review**

COVID-19 upended daily life and exacerbated global inequalities, resulting in about 6.88 million reported deaths globally by March 2023 (Johns Hopkins University, 2023) and 14.83 million excess deaths (i.e., the number of deaths during a crisis that exceeded the expected number under normal conditions, including those directly caused by the crisis and those resulting from indirect impacts like healthcare disruptions) (Msemburi et al., 2023). The development and regulatory approval of COVID-19 vaccines changed the trajectory of the pandemic. However, limited access to and acceptance of vaccination resulted in global vaccine coverage inequalities that continued to persist throughout the pandemic. Although 72.3% of the world’s population had received at least one dose of the COVID-19 vaccine as of March 2023 when many countries stopped regularly reporting vaccination data, there remained stark gaps in vaccination rates across regions (Holder, 2023). For example, 37% of the population received at least one dose of vaccination in Africa, 58% in the Middle East, 70% in Europe, 81% in the U.S. and Canada, and 82% in Asia-Pacific and Latin America (Holder, 2023). Among LMICs, surveys found that Europe and Central Asia showed the highest vaccine hesitancy (58%), followed by the Middle East and North Africa (47%), East Asia and Pacific (26%), Sub-Saharan Africa (15%), and Latin America and the Caribbean (8%) (Dayton Eberwein et al., 2023). In particular in LMICs, COVID-19 vaccination can reduce transmission and mortality, prevent healthcare system overload, support economic recovery, and address global vaccine inequity (Ali et al., 2022; WHO, 2021).

The journey to vaccination is a complex process involving structural factors, individual decisionmaking, and last-mile actions. The success of vaccination programs is determined by promoting vaccination intentions and translating these intentions into vaccine uptake. Issues of vaccine confidence (i.e., lack of trust in the vaccine or provider), complacency (i.e., perceived lack of need for a vaccine), and low convenience (i.e., limited access) can contribute to vaccine hesitancy<sup>3</sup> (Larson et al., 2014) and ultimately discourage vaccine uptake.

<mark>Communications campaigns offer one approach to strengthening vaccine confidence and promote intentions to get vaccinated. A</mark> lthough public health messages generally have small to moderate effects on attitudes and behaviors (e.g., Noar, 2006; Wakefield et al., 2010), these messages can have societal impacts as they reach large populations repeatedly (Funder & Ozer, 2019; Noar, 2006). A recen <mark>t summary of research found that media campaigns, both through traditional (e.g., television, radio, and printed materials) and new social media, can significantly impact health knowledge, attitudes, and behaviors</mark> (Orozco-Olvera & Malhotra, 2023). These approaches might be particularly effective when coupled with other behavioral and policy interventions such as vaccine certificates or automated vaccination appointment systems (e.g., van Bavel et al., 2020; Ruggeri et al., 2024).

As a key antecedent of vaccine uptake, it is important to understand the type of communication that fosters vaccine confidence and intentions to get vaccinated. Given the limited empirical evidence to guide such communication strategies as the pandemic was ongoing (Lim et al., in press), <mark>we launched a global research program of 28 online survey experiments leveraging social media across 23 mostly low-</mark>

> 3 Vaccine hesitancy refers to a “delay in acceptance or refusal of vaccines despite availability of vaccination services” on a continuum between full acceptance and refusal (WHO, 2014).

3
<mark>and middle-income countries and territories to help inform policy makers’ approach to communicate for promoting uptake of the COVID-19 vaccine.</mark> While research has explored vaccine acceptance globally, including new research on COVID-19 vaccine hesitancy, evidence on crafting effective communications to promote COVID-19 vaccine acceptance in low- and middle-income countries is still limited (Batteux et al., 2022; Xia & Nan, 2023). The gap in experimental evidence providing insights into effective messaging in LMICs and non-Western, Educated, Industrial, Rich, and Democratic (WEIRD) contexts remains notable. Additionally, existing syntheses of effective communication for encouraging COVID-19 vaccination were largely systematic reviews (e.g., Batteux et al., 2022; Xia & Nan, 2023) and evidence synthesis (e.g., Ruggeri et al., 2024; Nan et al., 2022; van Bavel et al., 2020) that did not use meta-analytic methods, which could identify overall effect sizes, moderators, and sources of heterogeneity (Card, 2012).

Our research complements this work by quantifying and comparing messages’ overall effect sizes and moderators across countries using meta-analytic methods, which can help generate more precise policy recommendations about effective messaging to promote vaccine confidence. COVID-19 vaccine hesitancy may differ from hesitancy toward long-established vaccines like measles, mumps, and Rubella (MMR); seasonal influenza; and diphtheria, tetanus, and pertussis (DTaP); or newer ones like human papillomavirus (HPV), due to its rapid introduction and promotion on a large scale (e.g., Dombrádi et al., 2021).

The purpose of this research is to summarize the work completed in support of policy makers to evaluate the effectiveness of different messages to increase COVID-19 vaccination intentions by summarizing the findings of 28 survey experiments with 123,720 adults recruited from social media in 23 countries and territories between January 2021 and June 2022 using two-stage individual patient data (IPD) meta-analysis (MA) methods. Results from the meta-analysis show that, among unvaccinated survey respondents, behaviorally informed messages significantly increased the odds of vaccination intention by 1.28 times overall. Significant pooled effects of specific framings ranged from increasing odds of vaccination intentions by 1.16 times (framing that highlighted the shifting nature of the pandemic and need to protect against different variants of the virus) to 1.45 times (framing that leveraged messenger effects and endorsement of the vaccine by experts and religious leaders). While messages featuring experts with religious leaders or celebrities, and dynamic social norms had consistent impacts, others, such as endemic, safety, experts, efficacy and pro-social, and variant messages, showed moderate-to-high heterogeneity. This research underscores the importance of communication tailored to address different drivers of vaccine hesitancy and offers insights for handling future health crises with behavioral communication strategies leveraging rapid insights afforded by social media.

While this research offers evidence to help close the gap in understanding how to effectively promote vaccine intentions in primarily LMICS, the study has limitations that should be noted. In particular, this research leverages non-representative Facebook sampling that under-represents older, poorer, and less educated groups, and could reflect self-selection bias. In addition, this study examines impact of messaging on self-reported intentions instead of actual vaccination behavior. Still, this work offers new evidence to fill gaps in understanding message effects on vaccine confidence in non-WEIRD and primarily low- and middle-income countries that could inform future research that aims to address this study’s limitations and future public policy efforts.

4
# **Literature Review**

As a part of pandemic responses, effective public health communication and messaging can help address vaccine hesitancy through strengthening confidence in the vaccine’s effectiveness and safety, and mitigating complacency about the potential risks of the target virus (Batteux et al., 2022; Nan et al., 2022; Xia & Nan, 2023). Who (i.e., messengers, information sources) says what (i.e., messages) are key factors influencing message effectiveness (Lasswell, 1948; McGuire, 1989). Using trustworthy messengers can help improve attitudes and encourage health behaviors (Huang & Sundar, 2022; Lu et al., 2021; Ruggeri et al., 2024). For example, during COVID-19, some studies found that vaccine uptake and vaccination intentions were increased when messages were conveyed by healthcare professionals (Bartoš et al., 2022), politicians (Pink et al., 2021), and messengers with religious orientation (Chu et al., 2021) and races (Alsan & Eichmeyer, 2024) aligned and/or matched to the target audience. However, other research has not found evidence for the messenger effect (Huang & Liu, 2022; Motta et al., 2021), underscoring the importance of testing messages and messengers in different contexts.

Regarding ‘what to say’ (content of the messages), prior research has identified promising message features applicable to encouraging COVID-19 vaccination including social norms, individual and collective benefits, and vaccine safety (Batteux et al., 2022; Ruggeri et al., 2024; Nan et al., 2022; Xia & Nan, 2023). In the COVID-19 context, some studies testing descriptive norms (i.e., what others do) in messages highlighting wide vaccine uptake increased vaccination intentions (Ryoo & Kim, 2021; Sinclair & Agerström, 2021), whereas other studies did not find effectiveness (Santos et al., 2021; Sasaki et al., 2022). Prior research found that messages focusing on individual benefits did not significantly increase COVID-19 vaccination intentions (Bokemper et al., 2021; Freeman et al., 2021; James et al., 2021), while messages focusing on collective benefits could increase the intentions (Freeman et al., 2021; James et al., 2021; Trueblood et al., 2022).

As some vaccine hesitancy reflected concerns over vaccine safety (Dayton Eberwein et al., 2023), messages focusing on vaccine safety have also been found to increase vaccination intentions (Argote Tironi et al., 2021; Ashworth et al., 2021; Davis et al., 2022; James et al., 2021; Palm et al., 2021). Lastly, to overcome the message fatigue (i.e., overexposed, redundant, exhausted, tedious) that people felt from prolonged, extensive COVID-19 vaccination communication (So et al., 2017) and consequent lack of engagement in protective behaviors (Guan et al., 2023), it is important to develop and use diverse messages (Okuhara et al., 2023). In our study, we developed messages focusing on evolving contexts, such as recognizing that the virus was becoming endemic and the growing number of new variants.

As noted, prior systematic reviews found that most experimental studies testing messages encouraging COVID-19 vaccinations were conducted in WEIRD societies and upper-income countries (Batteux et al., 2022; Xia & Nan, 2023), warranting research to test what messages can encourage vaccination behaviors in non-WIERD contexts. Given limited evidence from experiments testing COVID19 vaccine messages conducted in LMICs to address vaccine hesitancy, we aim to close this gap in the literature with a summary of insights from our global research program testing messages to encourage COVID-19 vaccinations across 23 non-WEIRD and mostly lower- and middle-income countries with metaanalyses.

5
# **2. Method**

## **2.1. Study Sample**

To identify whether different behaviorally informed vaccine messages can increase intent to vaccinate, we conducted 28 randomized survey experiments using social media in 23 countries and territories between January 2021 (when vaccines were not widely available) and June 2022 (when vaccine supply was widely available). See Appendix A for a list of sampled countries and when studies were implemented in each country. Multiple studies were conducted in Lebanon, Iraq, and Jordan throughout the course of the pandemic. Each study followed similar protocols for recruitment, experimental design, and survey measurement, which are described below. The results from individual studies have been reported in other publications for Belize, Haiti, Jamaica (Margolies et al., 2023), Papua New Guinea (Hoy et al., 2023), Saudi Arabia (Lim et al., 2024), and Zambia (Hoy et al., 2022). Also, a separate survey with family doctors in Romania (Badiani-Magnusson et al., 2022) was conducted as part of the same global research program, yet was not included in this IPD meta-analysis. Our experiments were approved by the Health Media Lab Institutional Review Board (HML IRB), approval number # 1017TWBG21.

## **2.2. Participant Recruitment and Sampling**

Participants were recruited through Facebook using quota sampling by displaying ads inviting participants to interact with a chatbot in Facebook Messenger to complete the survey (using languages of preference) for a chance to win a lottery reward.

The sample in each country was drawn from all Facebook accounts 18 years and older in the country. We used quota sampling by exposing the recruitment ads to pre-defined demographic groups based on age, gender, and region to mirror the country’s populations. To get a quota sample and optimize recruitment, we updated the ad placement in real-time based on participating Facebook users’ characteristics to ensure proper representation of each demographic group in the final sample using marketing applications such as the Virtual Lab (Rao et al., 2020) and RevealBot (RevealBot, 2024). For example, if a subgroup was underrepresented and had higher attrition or non-response, then the budget for ads targeting that demographic subgroup was automatically increased to better approximate the required sample for that subgroup.<sup>4</sup> Across all studies, the final sample included 229,886 responses in 28 studies with an average of 7,800 responses per online study. Only participants who had not received the COVID-19 vaccine were included in the experiment ( _n_ = 123,720), which is the focus of the results reported in this paper. No

> 4 For example, we recruited participants in Iraq targeting ads based on a combination of gender, age group, and governorate. This combination created x strata for recruitment with each stratum reflecting a combination of these three characteristics. For instance, one set of ads targeted women aged 40-49 who lived in the governorate of Baghdad while another set of ads targeted men aged 40-49 who lived in the governorate of Baghdad with quota targeting based on the latest available population data. During data collection for the study, if the proportion of participants from this demographic group in the study sample was lower than its proportion in the broader population, the recruitment ads were dynamically adjusted to spend more money to better target this population and increase its proportion in the study sample.

6
additional data cleaning was necessary because the Facebook Messenger chatbot ensures that only one survey is recorded per Facebook account and discards responses that deviate from the questions asked.

# **2.3. Study Procedures**

Participants who clicked on the study advertisement were redirected to Facebook Messenger, where the survey was conducted via chatbot. Participants selected their preferred language and gave informed consent at the beginning of the survey before proceeding to the survey modules.

First, we asked participants’ COVID-19 vaccination status. Those who were unvaccinated were included in the survey experiment that randomized participants to different messaging arms and asked for their vaccination intentions, our main outcome. We then asked participants about their perceptions of the main benefits of or concerns about COVID-19 vaccination, additional questions about attitudes and beliefs related to COVID-19, use of different information channels (e.g., social media, radio, etc.), and trusted information sources. Lastly, we asked participants to provide information about their demographics. The study took approximately 10 minutes, on average. See Figure 1 for a summary of participant inclusion and exclusion at each step of the studies.

**_Randomization._** After answering a question about their COVID-19 vaccination status, unvaccinated participants were randomized to study arms with different behaviorally-informed message framings about the COVID-19 vaccine. Randomization was stratified such that participants had a greater probability of being allocated to the control message compared to other arms, and all treatment message arms had an equal probability of assignment. See Appendix B for more detail on the proportion of samples randomized to each arm.<sup>5</sup>

> 5 More participants were assigned to the control condition to provide a subsample for diagnostics to understand attitudes and beliefs about a country’s social media population with potential impacts of framing. These diagnostics were used to inform the government’s COVID-19 strategy along with results from the survey experiments.

7
<!-- Start of picture text --> Ad . Facebook advertisement reach<br>vertisement (n = 39,295,292)<br>Began survey<br>(n= 229,886)<br>Excluded<br>- Not meeting inclusion criteria<br>- Declined to participate<br>Vaccination status<br>(n = 220,599)<br>M Vaccinated Unvaccinated<br>leasurement (n = 93,554) (n = 123,720)<br>Random Assignment<br>Allocated to the control group Allocated to treatment groups<br>(n = 62,046) (n = 61,674)<br>Vaccination intention<br>Measurement<br>Demographics<br><!-- End of picture text -->
context—for instance, messages relating to religious leader endorsement were tested in contexts where religion was expected to have greater impacts such as religious countries in the Middle East and North Africa. As such, direct comparisons of the magnitude of impacts across different framings should be interpreted with caution—as differences in impacts may reflect systematic differences in country and timeline of rollout. Two framings (financial incentives, mortal risk of COVID-19) were tested in fewer than three studies and were thus excluded from the meta-analysis. See Appendix C for the operationalization of message framings used in each study.

**Control framing (** **_k_ = 28,** **_n_ = 62,046).** The control framing was a pure control with no additional text before the vaccine intention question.

**Experts framing (** **_k_ = 16,** **_n_ = 14,152).** This framing aimed to leverage a messenger effect by emphasizing that experts consider the vaccines safe and effective (e.g., ‘ _The COVID-19 vaccines available in {country} ({type of vaccines}) are considered safe and highly effective by national and international experts’_ ).

**Experts + celebrities framing (** **_k_ = 7,** **_n_ = 9,492).** This framing leveraged the messenger effect by alluding to endorsements of experts as well as celebrities, who have also been demonstrated to have an influential effect on health behavior (Alatas et al., 2024; Chu et al., 2021) (e.g., ‘ _The COVID-19 vaccines available in {country} ({type of vaccines}) are considered safe and highly effective by national and international experts, and celebrities and famous athletes get it themselves_ ’).

**Experts + religious leaders framing (** **_k = 8_ ,** **_n_ = 10,082).** This framing leveraged the messenger effect by alluding to endorsement of religious leaders as well as experts, as religious leaders also play an important role in shaping health behaviors in many contexts ( <mark>Rakotoniana et al., 2014)</mark> (e.g., ‘ _If a COVID19 vaccine is considered safe and effective by national and international experts, and religious leaders in your community get it themselves, would you plan to take the vaccine?_ ’).

**Safety framing (** **_k = 12, n = 6,298)._** This framing contrasted the safety of the COVID-19 vaccines to the dangers of COVID-19 infection, addressing concerns about the safety of vaccination (e.g., ‘ _COVID19 vaccines are safe - there have been no reported hospitalizations in {Country} due to vaccinations compared to {number} deaths due to COVID-19’_ ).

**Efficacy and pro-social framing (** **_k = 14_ ,** **_n_ = 8,841).** This framing emphasized the efficacy of COVID-19 vaccines for protecting the vaccinated person and their family and friends from serious health complications. (e.g., ‘ _The latest studies from around the world confirm that the COVID-19 vaccines protect you, your friends and family from COVID-19 by substantially reducing or eliminating hospitalizations and deaths_ ).

**Dynamic social norms framing (** **_k = 12_ ,** **_n_ = 6,258)** This framing used messaging to convey that descriptive norms were shifting toward a growing number of people getting vaccinated against COVID19 (e.g., _‘{Nationality} are getting vaccinated against COVID-19! More than {number} have done it so far, with {number} just in the past 2 weeks alone’_ ). Messages used data available from governments or external sources to ensure the numbers provided were credible.

**Endemic framing (** **_k = 7_ ,** **_n_ = 3,215).** This framing emphasized the vaccination needs for the future, as the COVID-19 virus will continue to exist (e.g., _‘As we learn to live with COVID-19, it's likely we_

9
_will all be exposed to the virus eventually. The best way to prepare yourself is to get fully vaccinated because it greatly reduces the risk of hospitalization and death’_ ).

**Variants framing (** **_k = 6_ ,** **_n_ = 3,336).** This framing focused on the vaccine’s efficacy against new variants, to account for concerns at the time about whether the vaccines would be protective against different strains of the virus (e.g., _‘New variants like Omicron and Delta can be worrying, but the best evidence so far indicates that vaccines are still highly protective against serious illness and death from COVID-19’_ ).

# **2.4. Measures and Indices**

## **_Primary Outcome_**

**Vaccine Intention** **_._** The outcome of interest is self-reported vaccination intention, which was measured for respondents who reported that they were not yet vaccinated at the time of the study. Vaccine intention was measured as a categorical outcome by asking a question “ _Are you willing to/do you plan to get the COVID-19 vaccine_ ?” Response options included “Yes,” “Unsure,” and “No”. For our analysis, we recode these responses into a binary classification of “Yes” or “No/Unsure”.

## **_Potential Individual-Level Moderators_**

**Gender.** Participants were asked to indicate whether their gender was male, female, or other. In some country contexts, it was considered inappropriate to include the ‘other’ category, and participants were simply asked if they were male or female.

**Age.** Age was measured categorically, asking participants to indicate whether they were 18-29, 30-39, 40-49, 50-59, or 60+.

**Education.** Participants were asked to indicate the highest level of education they had completed – No Education, Primary Education, Secondary Education, or Tertiary Education.

**Health worker.** Participants were asked to indicate whether they were employed as a health worker or not.

## **_Potential Study-Level Moderators_**

Based on discussion with public health and behavioral science experts and a review of the literature, we explore the following potential moderators: the month of study launch, the GDP per capita of the study country, the vaccination rate in the country at the time of the study, and trust and vaccine perception measures from the Wellcome Global Monitor Survey (2018). The Wellcome Global Monitor survey is a survey of attitudes toward science and health based on a survey with over 140,000 people in more than 140 countries. Since these data from the Wellcome Global Monitor survey are measured at the individual level, we aggregate these measures to the country level by taking their mean value. This approach is consistent with practices in cross-cultural and health research (e.g., Gelfand et al., 2011; Weinberg et al., 2021), including research estimating generalized trust at the country level (e.g., Marozzi, 2014; Reeskens & Hooghe, 2008; Robbins, 2012). See Appendix D for more details on the empirical rationale for country-level aggregation of the Wellcome Global Monitor survey indicators. The Wellcome Global Monitor 2018 data were not available for Belize, Djibouti, Jamaica, Papua New Guinea, and West Bank and Gaza.

10
**Country-level vaccination rate (at least one dose).** We include an estimate of the proportion of the country’s population that received at least one dose of a COVID-19 vaccine. Country level data was obtained at the time of data collection from Our World in Data (2024), which collected data from public official sources. The effect of the intervention could be smaller in countries with higher vaccination rate because the remaining unvaccinated sample may be particularly reluctant to receive the vaccine. It might also reflect the level of deployment of the vaccine as well as the general acceptance of the vaccine.

**Intervention start date.** We included the start month of the data collection. The dates were centered so that the starting month of the first intervention (i.e., January 2021) was set to month 0, and so on until month 16, corresponding to the start of the last intervention (May 2022). The intervention start date might also reflect the available knowledge about the virus and availability of the vaccine at the time of the intervention.

**GDP per capita PPP.** We also explore country wealth using gross domestic product (GDP) converted to international dollars using purchasing power parity (PPP) rates from the World Bank (2024). An international dollar possesses equivalent purchasing power relative to GDP as the U.S. dollar does within the United States. The GDP per capita has been shown to correlate with a large number of social and health outcomes across countries (Bloom & Canning, 2000), which may influence the effectiveness of the behaviorally informed messages targeting vaccine intentions.  When including this variable in the study-level moderator analysis, we center the data by subtracting the median GDP per capita PPP of the sample countries from the GDP per capita PPP of each country.

**Trust in healthcare workers.** We explore whether trust in healthcare worders (e.g., doctors and nurses) before the onset of the pandemic moderates impacts of message framings. We use data from the Wellcome Global Monitor (Wellcome Trust & Gallup, 2018), which assessed trust in healthcare workers with an item rating “ _How much do you trust each of the following?_ ” for “ _Doctors and nurses in this country_ ” on a 1-4 scale, with response options of “A lot”, “Some,” “Not much,” “Not at all”. Respondents could also respond with “Don’t know,” and “Refused”—such responses were dropped from our analysis. For ease of interpretation, we reverse-coded the responses, so higher values indicate stronger trust such that 1 = _Not at all_ and 4 = _A lot_ . We aggregate trust to the country level by calculating the mean response within each country. We then center country-level trust measures by subtracting the median country-level trust measure from each country-level means.

**Perceived vaccine safety.** We included perceived vaccine safety before the COVID-19 pandemic. We used data from the Wellcome Global Monitor (Wellcome Trust & Gallup, 2018). Perceived vaccine safety was measured by asking “Do you agree, disagree, neither agree nor disagree with the following statement? Vaccines are safe.” on a 1-5 scale, with response options of “Strongly agree,” “Somewhat agree,” “Neither agree nor disagree,” “Somewhat disagree,” “Strongly disagree,” “Don’t know,” and “Refused.” We reverse-coded the responses, so higher values indicate stronger agreement, such that 1 = _Strongly disagree_ and 5 = _Strongly agree_ . We aggregate perceived vaccine safety to the country level by calculating the mean response within each country. We then center country-level vaccine safety perceptions by subtracting the median country-level trust measure from each country-level mean.

**Perceived vaccine effectiveness.** We included perceived vaccine effectiveness before the COVID19 pandemic using data from the Wellcome Global Monitor (Wellcome Trust & Gallup, 2018). Perceived vaccine effectiveness was measured by asking “Do you agree, disagree, neither agree nor disagree with

11
the following statement? Vaccines are effectiveness.” on a 1-5 scale, with response options of “Strongly agree,” “Somewhat agree,” “Neither agree nor disagree,” “Somewhat disagree,” “Strongly disagree,” As with the measure of vaccine safety, we reverse-coded the responses, so higher values indicate stronger agreement, and responses of “Don’t know” and “Refused” were dropped. We aggregate perceived vaccine effectiveness to the country level by calculating the mean response within each country. We then center country-level vaccine safety perceptions by subtracting the median country-level trust measure from each country-level mean.

# **2.5. Meta-Analytic Approach**

To summarize findings from our research program, we conduct a meta-analysis using a randomeffects, two-stage individual participant data (IPD) meta-analytic approach (Riley et al., 2021). The random effects model allows for variation in study effects, appropriate for our context where operationalizations of independent variables and outcome measurements differ slightly across studies. Moreover, IPD enables us to fully leverage the individual level data collected, allowing us to explore study-level moderators that could explain variation in messaging effects across studies as well as heterogeneity in treatment effects that could be explained by individual-level characteristics. IPD metaanalyses are commonly used in contexts such as clinical research, where individual participant data is more readily available, offering advantages over the traditional meta-analytic approach that rely solely on aggregate statistics (Burke et al., 2016; Debray et al., 2015; Tierney et al., 2015). As noted earlier, the operationalization of message framings and the primary outcome were tailored to each study’s context. All analyses were conducted with Stata (version 18).

To estimate the treatment effects for the framing experiment, we use logistic regression with treatment arm as the exposure and a binary measure of vaccine intention as the outcome variable (1 = intend to vaccinate). In our analysis, we control for a range of individual-level covariates, including gender, age, education, and whether the participant was a health worker. For covariates and moderators, we use categorical variables coded to have a category for non-response (i.e., missing) to avoid dropping observations where possible. In the first stage of the IPD MA, we calculate the logit model for each one of the studies as follows: 1 (1)<sup>ii+ γii+ )</sup> For s =1,2,3...,28 studies pp(xx) = 1+ee<sup>−(β</sup> 12
We consider pooled effects to be significant when the 95% confidence intervals do not include 0 in their range.

For heterogeneity, we use the _I_<sup>_2_</sup> index, or the ratio of excess to total variability, to measure inconsistency of effect sizes. To interpret the _I_<sup>_2_</sup> index, 25 is generally considered low, 50 is moderate, and 75 is high (Higgins, 2003).

# **_Missing Data Analysis_**

We conduct Little's Missing Completely at Random (MCAR) test to determine whether missing data in our datasets are MCAR –meaning, whether missingness is unrelated to any observed or unobserved data. Little’s MCAR test showed that the missing data in gender, education, age and health worker status are not MCAR in 19 of the studies (see Appendix E). While the studies conducted in the Facebook Messenger chatbot required participants to answer each question before moving on, there are other reasons for missing data. First, the three studies in Tunisia and the study in Ukraine were conducted in SurveyMonkey and allowed participants to skip questions. Second, some studies added the response options "other" or "prefer not to answer" to demographic questions that were coded as missing due to small sample sizes. Third, in some cases, at the request of the in-country counterpart(s) or to assess whether demographic questions increased attrition, these questions were asked after the experiment, resulting in valid observations with no responses to demographic questions. Finally, some questions were not included at the request of the country counterpart(s), particularly those related to health worker status.

Following best practices non-random missingness, instead of analyzing only complete cases, we use the model-based multiple imputation for our analysis (Allison, 2002; Enders, 2022; Schafer & Graham, 2002) with ordered logistic regression, creating 18 imputations and randomly sorting the dataset. Although multiple imputation assumes that data is Missing at Random (MAR), it can often remain unbiased when data is Missing Not at Random (NMAR) (Schafer & Graham, 2002). This imputation method estimates multiple values based on the distribution of the observed data to capture uncertainty. Multiple imputation involves three steps: first, the missing data is filled in with estimated values _m_ times to create _m_ complete datasets; second, each of these datasets is analyzed using a statistical method; finally, the results and parameter estimates are pooled for inference. Following recommendations that _m_ should equal the percentage of incomplete cases (Bodner, 2008; White et al., 2011), we used 18 iterations, with missing cases ranging from 1% to 17%. Multiple imputation provides more accurate results by considering the variability from both sampling and imputation.

# **_Individual-Level Moderator Analysis_**

To analyze individual-level moderators, we perform subgroup analyses for gender, age, education level, and health worker occupation. Interactions between treatment and gender, treatment and age, treatment and education level, and treatment and occupation are evaluated by logistic regression analyses. For ease of interpretation, we explore interaction effects be recoding the age and education categorical variables as binary. When interactions were statistically significant, further post hoc analyses are performed with participants of male vs. female, 40 years old or older vs. younger than 40 years old, secondary education or below vs. tertiary education, and health workers vs. non-health workers. More specifically, we estimate the pooled effects within each subgroup of interest to 13
decompose significant interactions, following the same procedure as for main effects but only including observations of the relevant subgroup.

### **_Study-Level Moderator Analysis_**

Study-level moderator analysis is conducted for framings where significant study-level variation is observed according to the _I_<sup>_2_</sup> index. Specifically, random-effects meta-regressions where study-level moderators are regressed on the effect sizes of a given framing across different studies are performed, with a meta-regression estimated for each framing. In this model, regression coefficients indicate the relationship between a covariate and the impact of a given framing. For example, we can interpret a regression coefficient of 0.03 for intervention start date as indicating that an increase in one month of the start date of a study corresponds to an increase of 0.03 units in log odds ratio in the impact of the experts framing on vaccination intentions.

Given the number of explored moderators (7) and the small number of studies testing certain framings (i.e., endemic, variants), we first examined the bivariate correlation between moderators and effect sizes and included only those with a significant relationship in our final meta-regression model for a given framing. While other IPD MA studies also had small numbers of studies (i.e., 6-7) when conducting moderator analysis (e.g., Jenum et al., 2019; Louise et al., 2021; Smelt et al., 2018), we note that our results should be interpreted with caution as we have a modest number of 6-16 studies included in our analyses to explain heterogeneity across studies. For efficacy and pro-social ( _k_ = 14), endemic ( _k_ = 7) and variant ( _k_ = 6) framings, none of the tested moderators explained a significant amount of variation in effect sizes.

# **3. Results**

## **3.1. Characteristics of Participants and the Study Sample**

Table 1 presents the demographic breakdown of the study sample, including the total sample that completed the surveys—both vaccinated participants who did not participate in the survey experiment and unvaccinated participants who were included in the survey experiments. Reflecting the typical social media user in LMICs, the sample skewed toward being male (52.4%), highly educated (43.6% tertiary education), and somewhat young (29.2% 18-29 years old).

14
_Table 1. Summary of individual-level characteristics of full sample and survey experiment sample included in the meta-analysis._

|**Variable**|**Group**|**Total**<br>**(****_n_=217,254)**|**Unvaccinated**<br>**Experiment**<br>**Sample**<br>**(****_n_=123,720)**|**Vaccinated**<br>**Sample**<br>**(****_n_=93,554)**|
|---|---|---|---|---|
||Male|52.35|51.64|53.30|
|Gender|Female|47.15|47.80|46.29|
||N/A|0.50|0.56|0.41|
||18-29|29.18|33.34|23.68|
||30-39|21.78|22.90|20.29|
|Ae|40-49|17.68|16.04|19.84|
|g|50-59|14.20|10.58|19.00|
||60+|6.68|4.41|9.68|
||N/A|10.48|12.73|7.51|
||No education|1.43|1.40|1.48|
||Primary|8.02|8.00|8.05|
|Education|Secondary|34.78|33.69|36.22|
||Tertiary|43.57|42.22|45.35|
||N/A|12.20|14.70|8.90|
||No|70.51|70.21|70.91|
|Health Worker|Yes|12.27|12.83|11.52|
||N/A|17.22|16.96|17.57|

_Note._ Distribution of respondents by key demographic characteristics and COVID-19 vaccination status. N/A includes participants who chose this response option or did not complete the survey. Gender, age, and education were asked in all surveys, whereas health worker status was missing in The Gambia.

Given the large size of the sample across all studies, all differences in the demographic characteristics between the unvaccinated and vaccinated samples are significant. Qualitatively, however, the sample characteristics are similar between unvaccinated and vaccinated in gender and education, with slightly larger proportions of more educated participants in the vaccinated sample. The biggest difference between the unvaccinated sample used for the intervention and the vaccinated sample is the age distribution, potentially reflecting most countries’ policies of vaccinating the older population first. Unexpectedly, however, the proportion of participants indicating that they were health workers was slightly greater among the unvaccinated (12.8%) than vaccinated (11.5%) samples.

The studies were conducted over 18 months in mostly low- and middle-income countries and territories, reflecting demand from policy makers for more evidence to guide their communication strategies. Table 2 summarizes the timeline of implementation across different studies and the corresponding rate of vaccination at time of data collection by region (see Appendix A for statistics for each study). Early on, implementation of the studies began before any vaccines were distributed including Lebanon Round 1 (R1), Iraq R1, West Bank and Gaza, Tunisia R1, Libya, Chad, Cameroon, and Honduras. Towards the end of the research program, studies were implemented in some contexts where about half the country population or more had been vaccinated (Saudi Arabia, Tunisia R3, 15
Lebanon R2, Belize). Given the wide range in timing of studies and vaccination rates, we take into account these differences in our moderator analysis examining study-level heterogeneity.

Our samples recruited from social media reported significantly higher vaccination levels than the national average at the time of the survey in all studies. This suggests that the unvaccinated population in countries with higher coverage already had access to the vaccine (compared to countries with low vaccine access) and, therefore, it is plausible that the unvaccinated in these cases were particularly resistant to vaccination. In addition, this finding highlights some of the potential uniqueness of our study samples—these samples recruited through social media are not representative of the general population, and results should be interpreted with this in mind.

_Table 2. Summary of study-level moderators of full sample and survey experiment sample included in the meta-analysis._

|**_Region_**|**_No. of_**<br>**_Studies_**|**_Start dates of_**<br>**_Data Collection_**|**_Mean_**<br>**_COVID-19_**<br>**_vaccination_**<br>**_rate in_**<br>**_sample_**|**_Mean_**<br>**_National_**<br>**_COVID-19_**<br>**_vaccination_**<br>**_rate at_**<br>**_time of the_**<br>**_experiment_**|**_Mean_**<br>**_GDP per capita,_**<br>**_PPP_**<br>**_(constant 2017_**<br>**_international $)_**|**_Mean_**<br>**_National_**<br>**_Trust in_**<br>**_doctors_**<br>**_and nurses_**<br>**_(1-4)_**|**_Mean_**<br>**_National_**<br>**_Perceived_**<br>**_vaccine_**<br>**_safety_**<br>**_(1-5)_**|**_Mean_**<br>**_National_**<br>**_Perceived_**<br>**_vaccine_**<br>**_effectivenes_**<br>**_s_**<br>**_(1-5)_**|
|---|---|---|---|---|---|---|---|---|
|Middle East & North<br>Africa|14|Jan-21 to Apr-22|37%<br>(0%, 86%)|22%<br>(0%, 54%)|16,655<br>(4,768; 47,315)|3.0<br>(2.7, 3.2)|4.5<br>(4.3, 4.6)|4.5<br>(4.4, 4.7)|
|Sub-Saharan Africa and<br>the Sahel|5|Aug-21 to May-22|39%<br>(0%, 72%)|9%<br>(0%, 19%)|2,367<br>(1,060; 3,757)|2.9<br>(2.5, 3.4)|4.3<br>(4.1, 4.7)|4.5<br>(4.4, 4.7)|
|Europe, Asia & Pacific|5|Jun-21 to Dec-21|45%<br>(16%, 64%)|21%<br>(1%, 39%)|12,630<br>(3,948; 18,307)|3.0<br>(2.9, 3.1)|3.9<br>(3.3, 4.2)|4.1<br>(3.8, 4.4)|
|Latin America &|4|Mr-21 t Ar-22|41%|22%|6,997|3.0|4.3|4.6|
|Caribbean||a o p|(0%, 89%)|(0%, 59%)|(3,102; 10,244)|(3.0, 3.1)|(4.1, 4.6)|(4.5, 4.6)|

_Note_ . Experiment start date is the month of Facebook ad launch for the first and last intervention in the region. COVID-19 vaccination rate and GDP per capita show the mean for the region with the minimum and maximum in parenthesis.

16
# **3.2. Main Effects on Vaccine Intentions**

We first explored the overall effect of behaviorally-informed framings compared to the control group. The pooled estimate of all tested messages’ impacts on vaccination intentions was a log odds ratio [LOR] of 0.24 (95% CI: [0.22, 0.27], OR: 1.28), indicating that he use of any behaviorally-informed messaging increased participants’ odds of having the intention to get vaccinated against COVID-19 by 1.28 times compared to the control group (see Appendix F for meta-analysis results transformed into odds ratios).

Next, we estimated the effect of each framing on vaccination intentions—see **Figures 2 and 3** for the study level effect by message framings compared to the control group. In descending order of magnitude, the pooled estimates of the messages’ with significant<sup>6</sup> impacts on vaccination intention was a 0.37 LOR (95% CI: [0.32, 0.42], OR: 1.45) for the message using experts and religious leaders, 0.35 LOR (95% CI: [0.27, 0.44], OR: 1.42) for the endemic framing, 0.28 LOR (95% CI: [0.23, 0.33], OR: 1.33) for the message using experts and celebrities, 0.24 LOR (95% CI: [0.18, 0.29], OR: 1.27) for the efficacy and pro-social framing, 0.23 LOR (95% CI: [0.19, 0.27], OR: 1.26) for the message using experts, 0.15 LOR (95% CI: [0.06, 0.24], OR: 1.16) for the variants framing, and 0.14 LOR (95% CI: [0.07, 0.20], OR: 1.15) for the dynamic social norms framing, meaning that these message framings were effective. Conversely, no difference between the safety message group and the control group was observed from the pooled estimates (LOR: 0.04, 95% CI: [-0.02, 0.11], OR: 1.05).

Messages using experts and religious leaders ( _I_<sup>_2_</sup> = 39.5%, _p_ = 0.115), experts and celebrities ( _I_<sup>_2_</sup> = 0.0%, _p_ = 0.686), and dynamic social norms ( _I_<sup>_2_</sup> = 0.0%, _p_ = 0.489) showed non-significant levels of heterogeneity, meaning that these messages showed a consistent level of impacts across studies and countries.

Conversely, results indicated significant moderate-to-high heterogeneity across the studies in the impact of the endemic framing ( _I_<sup>_2_</sup> = 75.7%, _p_ < 0.001), the safety framing ( _I_<sup>_2_</sup> = 70.1%, _p_ < 0.001), the experts framing ( _I_<sup>_2_</sup> = 69.4%, _p_ < 0.001), the efficacy and prosocial framing ( _I_<sup>_2_</sup> = 60.3%, _p_ = 0.002), and the variant framing ( _I_<sup>_2_</sup> = 60.7%, _p_ = 0.026), indicating that that these messages’ impacts significantly vary by study.

Specifically, the endemic messages significantly increased vaccination intentions in Iraq, Lebanon, and Haiti, but not in Tunisia, The Gambia, Belize, and Jamaica. The safety messages were effective only in Papua New Guinea and Serbia, but not in Kuwait, Jordan, Saudi Arabia, Djibouti, Tunisia, Jordan, the Republic of Congo, The Gambia, Kosovo, and North Macedonia. The messages using experts were effective in Lebanon, Iraq, West Bank and Gaza, Tunisia, Libya, Chad, Cameroon, Serbia, and Honduras, but not in Kuwait, Jordan, Saudi Arabia, Djibouti, Papua New Guinea, Kosovo, and North Macedonia. The efficacy and prosocial messages were effective in Jordan (R1 and R2), Zambia, Kosovo, Servia, and Haiti, but not in Kuwait, Saudi Arabia, Djibouti, Lebanon, North Macedonia, Ukraine, Belize, and Jamaica. Lastly, the variant messages significantly increased vaccination intentions in Iraq and Haiti, but not in Tunisia, Lebanon, Belize, and Jamaica.

> 6 When reporting results, we use the term 'significance' to imply statistical significance.

17
<!-- Start of picture text --> Experts<br>No. log Odds ratio x<br>Study pls 195% Cl) Weight<br>Lebanon RI 3137 0.27 (0.09, 0.44) 5.36<br>lrag Rt 6916 0.27 (0.18040) 13.47<br>West Bankand Gaza 8691 031 (0.21,041) 16.09<br>Tunisio 1 8565 0.22 (0.11,0.32) 15.00<br>Ubyo 4732 0,30 (0.16, 0.43) all<br>Kuwait V3 : 0.08 (-0.38,0.21) 1.88<br>vordon 8) “7277 ': 05 (-0.19, ;<br>Soudl Arabia 445, - 0.050.22 (-0.19,0.08)(-0.74,0.29) 8.930.63<br>Djooutt N50 -: 0.02 (032,027) 1.95<br>Chad 2619 oe 0.25 (0.04, 0.46) 3.84<br>‘Cameroon 1834 + 0.43 (0.20,0.65) 3.36<br>PopuaNew Guinea 1157 + 0.15 (0.16045) 1.79<br>Kosovo 2572 4 0.15 (003,033) $12<br>SerbiaNorth Macedonic 29361183 SinanH' 0.310.12(0.11,0.50)(0.40,0.15) 4.342.26<br>Honduras 3739 'a 0.42 (0.27,(0.27, 0.58) 0.58) 6.88<br>Overall Iv 99762 © 0.23 (0.19027) 100.00<br>("= 69.48, p< 0.001)<br>1 oO J<br><!-- End of picture text --> <!-- Start of picture text --> nae<br>Experts + Celebrities<br>No. log Odes ratto 8<br>seedy seal (eR cH) Wolgnt<br>Lebanon RI 3236 + 0.25 (0.08, 0.42) a2<br>Foo RI 657 — 0.23 (0.12. 0.94) 19.40<br>Wes! Bank and Gaza e708 ~~H 0.29 0.19.0.) v0<br>Tunlsiovane R1 8420 Seam' 0.31 ‘(0.20, 0.41); 21.80<br>lbys 4728 — 0.22 10.08. 0.35) 12.93<br>‘Cameroon 1225 + 0.38 (0.15, 0.60) 467<br>Honduras 3724 + 0.37 10.22, 0.53) 981<br>Overait1V a8 © 0.28 (0.23, 0.33) 100.00<br>( . =00% p= 0.688)<br>M © 1<br><!-- End of picture text --> <!-- Start of picture text --> Experts + Rel i cigious leaders<br>No. log Odds ratio ®<br>Study pis (95%. CN) Weight<br>Lebanon RI aise —_ 0.44 (0.27, 0.62) 782<br>rage 6938 — 0.35 10.23, 0.46) 13.22<br>West Bank and Gaza 8781 ~~ 0.43 (0.33, 0.53} 22.02<br>Tunisia RI 8677 > 0,24 (0.14, 0.34) 2091<br>Libya 4738 — 0.38 (0.25, 0.52) 12.18<br>Ched 2671 + 0.44 (0.24, 0.65) 543<br>‘Cameroon 1825 + 0.54 (0.32, 0.76) 451<br>Honduras ar34 —_ 0.36 (0.20, 0.51) 9.20<br>‘Overol, IV 40522 © 0.37 (0.32, 0.42) 100.00<br>(?=39.5%,p=0.115)<br>1 ° '<br><!-- End of picture text --> <!-- Start of picture text --> Safety<br>N ° log ee Odds cacsraeratis %<br>Study pts (95% Cl) Weight<br>Kuwait 1748, - -0.10 [-0.39, 0.19) 493<br>Jordan R1 4430 0.03 (0.10, 0.18) 22.39<br>Saudi Arabia a7 . 0.02 [-0.57, 0.52) 1.35<br>Djooutl 1104 - 0.06 (0.22, 0.39) 440<br>Tunisia R2 4952 ' 0.15 (-0.29, 0.01) 19.93<br>Jordan R2 174) - 0,08 [-0.30, 0.14) 834<br>‘Congo: 540 + 0.40 (0.01, 0.81) 2.38<br>Gambia 408 0.07 (-0.50, 0.35) 2.24<br>Papua New Guinea 1123 ! —F—__ 0.46 (0.37, 0.95) 476<br>Kosovo 2568 ; 0.02 [-0.20, 0.14) 12.43<br>Seria eer '—i— 0.28 (0.09, 0.48) [oar<br>North Macedonia 1215 - 0.15 (0.10, 0.41) 6.18<br>Overall, IV 23213 c> 0.04 [-0.02, 0.11) 100.00<br>(= 70.1%, p< 0.001)<br>A 0 1<br><!-- End of picture text -->
<!-- Start of picture text --> Efficacy and Pro-Social<br>No. loglog Odds Odds rat ratio %<br>study pts 1958CH Weight<br>Kuwoit 1775 0.13 (0.17,0.42) 3.44<br>Jordan RI 2408 0.21 (0.08,0.35) 16.25<br>Saudi Arabia 429 n -0.32 (0.85, 0.20) ww<br>Djibouti 1132 0.16 (-0.13, 0.45) 3.42<br>Jordan RZ 1735 —— 0.30 (0.08, 0.52) 6.40<br>Lebanon R2 4223 0.11 (-0.06, 0.28) 10.53<br>Zombia 2924 ‘ a 0.46 (0.31, 0.62) 13.31<br>Kosovo 2575 0.24 (0.06, 0.42) 9.45<br>Serbia 2915 7 0.33 (0.14, 0.53) 789<br>North Macedonia 162 0.25 (-0.02,0.52) 4.19<br>Ukraine 707 — 0.34 (-0.06, 0.75) 1.84<br>BelizeHaiti 6878255 - 0.30-0.12 (0.17,(0.99,0.44)0.75) 16.710.40<br>JamaicaOverai=: 60.3%,IV p= 0.002} 3269015872 ©'| 0.240.24 (0.18,(0.49,0.29)0.01) 100.004.86<br>4 ° 1<br><!-- End of picture text -->

<!-- Start of picture text --> Dynamic Social Norms<br>No. log Odds ratio %<br>Study: pls {95% Cl) Weight;<br>,<br>Kuwait 1750 0.05 |-0.25, 0.35) 47)<br>Jordan RI 4355 0.10 |-0.03, 0.24) 2202<br>Seudi Arabia ay + 0.50 {-0.13, 1.12) 1.07<br>Djibouti 1136 i" 0.27 (0.01, 0.56) 5.10<br>Tunisia R2 5036 4 0.05 [-0.10, 0.19) 2077<br>Jordan R2 1716 0.18 [-0.04, 0.41) 837<br>‘Congo 544 * 0.02 [-0.45, 0.40) 231<br>Papua New Guinea Noz 0.11 |-0.21,0.42) 418<br>Rosove 208 0.15 (003,033) 1277<br>Serbia 2862 ! + 0.25 (0.05, 0.48) 9.90<br>Notth Macedonia 1203 ‘| 0.08 |-0.17,0.35) 6.15<br>Overall,Ukraine IV 2341374 >elpa 0.550.14 (0.15,(0.07, 0.20) 0.94) 100.002.66<br>("= 0.0%,p =0.489)<br>a 0 1<br><!-- End of picture text -->

<!-- Start of picture text --> Endemic<br>Study No. log Odds ratio ®<br>pts (95%. Ch Weight<br>Tunisiaroa k? Ro 20681163 ———A: 0.1907 (0.33, (-0.12.049) 082) si768<br>Lebanon k2 420 ; 2 0220 (018, 0.46) 25.66<br>‘Gombia 414 0.07 (-0.35, 0.48) 4al<br>Halt 678) ‘4 0.51 (0.37, 0.65} 37.92<br>BelizeOverahJamaica(f=. 75.7%. IV p <0.001) 164881588254 >i:'‘ 0.730.060.35 (0.27.(-0.32.(-0.04, 0.44)1.50)0.16) 100.00Wae71.20<br>-l o<br><!-- End of picture text --> <!-- Start of picture text --> Variants.<br>No. log Odd ratio ®<br>Study pts (95% Cl vein<br>Tunisia R3 1414 , 0.10 :(0.14, 0:35) 12.50<br>roa R2 2061 +j 0.29 (0.03. 0.55) 10.95<br>Lebanon R2 4226 ‘ 0,00 (-0.17, 0.18) 24.45<br>Hit o7e9 ‘‘ 0.20 (0.16. 0.44) 38.29<br>BelizeOveraitJamaica[?. = 60.7%. 1V p = 0.026) Peery1598245 oO:4 0.1701100.15 (0.06, (-0.34.0.14) (-0.48,0.24)1.93) 100.001279Lol<br>1 ° 1<br><!-- End of picture text -->
The impacts of endemic framing are marginally larger in interventions conducted later than earlier ( _β_ = 0.046, _p_ = 0.057), but only when controlling for Wellcome indicators. Specifically, on average, for each month increase of the intervention start date, there is a marginally corresponding increase of 0.046 units in the impact of endemic messages on vaccination intentions.

# **_Country-Level Trust in Healthcare Workers_**

Trust in healthcare workers (e.g., doctors and nurses) was not a significant moderator for any of the framings.

# **_Country-Level Perceived General Vaccine Safety_**

General perceived vaccine safety (i.e., not COVID-specific) was not a significant moderator for any of the framings.

# **_Country-Level Perceived General Vaccine Effectiveness_**

The impacts of variants framing ( _β_ = 1.21, _p_ = 0.096) are marginally larger in countries with higher general perceived vaccine effectiveness. Specifically, on average, for each unit increase in level of agreement with the statement “Vaccines are effective”, there is a marginally corresponding increase of 1.21 units in the impact of variants framing on vaccination intentions.

_Table 3. Study-level moderation meta-regression results._

|Moderators|Exp|erts<sup>a</sup>|Safety|Efficacy<br>and Pro-<br>Social|End|emic<sup>a</sup>|Variants|
|---|---|---|---|---|---|---|---|
||(1)|(2)|||(1)|(2)||
|COVID-19 vacc. rate in sample||||0.003|-0.000|-0.004||
|||||(0.002)|(0.005)|(0.004)||
|Start month of experiment|0.031**|0.025||0.005|0.021|0.046*||
||(0.013)|(0.016)||(0.011)|(0.021)|(0.024)||
|Trust in healthcare workers||-0.368|-0.421|||||
|||(0.405)|(0.340)|||||
|General perceived vaccine safety|||-0.248<br>(0.207)|||||
|General perceived vaccine||||||-0.774|1.213*|
|effectiveness||||||(1.363)|(0.729)|
|Observations|16|13|10|14|7|5|4|
|Wellcome Global Monitor<sup>a</sup>|No|Yes|Yes|No|No|Yes|Yes|
|_I_<sup>2</sup>|86.0|85.6|37.4|67.7|78.3|71.2|72.0|

_Note:_ * _p < .10, ** p < .05. a_ Because of missing data with trust in doctors, perceived vaccine safety, and perceived vaccine effectiveness, for which data from the Wellcome Global Monitor 2018 were not available for Belize, Djibouti, Jamaica, Papua New Guinea, and West Bank and Gaza, we report the results of two study-level moderator meta-regressions for experts and endemic framings: (1) without the Wellcome Global Monitor indicators and all countries and (2) with the Wellcome Global Monitor indicators, dropping countries with missing indicators.

20
<!-- Start of picture text --> Female (base=Male)<br>Experts e ot) @<br>Experts + Celebrities Ooatho<br>Experts + Religious leaders ® COmoe<br>Safety @ @ joo<br>Efficacy and Pro-Social oo baaeao<br>Dynamic Social Norms ° O@ Dc ° °<br>Endemic @® CE®@<br>Variants cog 0 CO<br>-16 -1.2 -08 -04 00 04 08 41.2 1.<br>log Odds ratio<br><!-- End of picture text --> <!-- Start of picture text --> Age 40+ (base=Age 39-)<br>Experts e © emmjo00 ©<br>Experts + Celebrities © cajco<br>Experts + Religious leaders e OPCO 00<br>Safety ° wo ® |o eo<br>Efficacy and Pro-Social @® O@cD °<br>Dynamic Social Norms @Moo0pmoo o<br>Endemic ° a e °<br>Variants ° @o<br>-16 -1.2 -08 -04 00 04 08 41.2 1.<br>log Odds ratio<br><!-- End of picture text -->

<!-- Start of picture text --> Secondary or below (base= Tertiary)<br>Experts COOGHEEID<br>Experts + Celebrities mp Oo<br>Experts + Religious leaders Oo capa<br>Safety Oo oOofpMo0O fome)<br>Efficacy and Pro-Social e@ © @@moo00<br>Dynamic Social Norms o@ OM0® e e<br>Endemic a @ o<br>Variants o|}@ @o<br>-16 -1.2 -08 -04 00 04 08 41.2 1.4<br>log Odds ratio<br><!-- End of picture text -->

<!-- Start of picture text --> Health worker (base= rest)<br>Experts 4 @ ° codammao aD<br>Experts + Celebrities oo @<br>Experts + Religious leaders @ho<br>Safety coomop Oo Oo<br>Efficacy and Pro-Social @ 0G@)@oo<br>Dynamic Social Norms] @0 © 0 0O@/O 00 @<br>Endemic e o oO ° °<br>Variants e 00 0<br>-1.6 -1.2 -08 -04 00 04 08 1.2 1.6<br>log Odds ratio<br><!-- End of picture text -->
~~ood~~

# ~~<mark>sss</mark>~~

<!-- Start of picture text --> Age 39-<br>study pls (95% Cl) Weight<br>Lebanon R1 588 + 0.22 (0.08, 0.57) 438<br>WestBankandGaza 5412 —— 0.32 (0.19, 0.45) 5.77<br>bya 2938 — 0.27 (0.09, 0.44) 89<br>jordan RI 2626 ; 0.11 (0.07,0.28) 8.33<br>Soudl Arable 290 1 0:32 (095,031) 0.66<br>Djiboutl 033 - 0.09 (0.210.480) 2.84<br>Chad 236 + 0.26 (004,048) 5.48<br>Comeroon 593 —— 0:38 (0.14062) 4.56<br>PopuaNewGuhea 95 - 006 (0.29,0.41) 2.16<br>Serbia 484 + 0.40 (0.12,0.68) 3.32<br>North Macedonia 46 ' 0.26 (061,008) 2.24<br>jonduras 2352 —i 0.47 (0.28, 0.66 2<br>Overall, V 35436 ro) 0.26 (021,031) 100.00<br>(= 55.7%, p = 0.004)<br>1 ) 1<br><!-- End of picture text --> <!-- Start of picture text --> Age 40+<br>Study pls (95% Cl) Weight<br>Lebanon R 549 4 0.22 (0.04,0.4 68<br>West BankandGaza 3279 a 0.30 (0.14, 0.46) 16.87<br>Joya 1789 a 0.36 (0.14, 0.58) 917<br>Jordan R 746 — || 0.29 (-0.50,-0.08) 9.96<br>Saud! Arabia 55 1 0.03 (0.85,0.9 os?<br>Djibout y ——>a — _ | 10 (-2.07,-0.13) 0.47<br>Choa 252 5 0.23 (0.39,.0.86) 4<br>Cameroon 24 S$. 07 017,137) 22<br>PapuaNewGuines 200 - 0.48 (0.17, 1.14) 05<br>Serble 452 0.20 (0.07, 0.4 6.05<br>North Macedonia 43 : 0.13 (0.32,0.58) 2<br>jonduras 38: —_ 0.35 (0.08, 0.6 6.28<br>Overall, IV 2032 > 0.18 (0.11,0.25) 100.00<br>(= 61.0%,p <0.001)<br>0 1<br><!-- End of picture text -->
<!-- Start of picture text --> Age 39-<br>No. log Odds Odds ratio. %<br>Study pts (95% cl) Welght<br>Kuwail 986 0.12 (-0.28, 0.51) 0.51) 383<br>Jordan RI 2674 0.11 (0.06, 0.28) 20.28<br>Soudi Arabia Arabia 283 + 0.20 (-0.95, 0.35) 143<br>Djibouti 993 i 0.14 [-0.17, 0.46) 0.46) 601<br>Tunisia R2 R2 317 ' 0.10(0.27,007) 2088<br>Jorcian R2 R2 1180 - 0.02 [-0.24, 0.28) 0.28) 877<br>Congo a +__ 0500008, 094) 094) 3.12<br>aria ass 0705203 301<br>Papua New Guinea New Guinea Guinea v2 : —_—@£$. 07 (045, 1.10) 573<br>Kosovo W97 - 0.02 [-0.19, [-0.19, 0.23) 1377<br>Sorbla 1518 —— 0.34 (0.07, 0.62) 0.62) 7.80<br>North Macedonia Macedonia 751 ~ 0.15 (0.48, 0.19) (0.48, 0.19) $37<br>Overt, IV IV 15237 > 0.09 (0.01, 0.17) 0.17) 100.00<br>(= 66.6%, p < 0.001)<br>| 0 1<br><!-- End of picture text -->

<!-- Start of picture text --> Age 40+<br>No. log Odds Odds ratio %<br>Study pis (95%CH Weight<br>Kuwait 762 0.44 (0.76, 0.08) 693<br>Jorcian RI RI 1756 0.09 (-0.30, 0.12) 0.12) 27.03<br>Saudi Arabia Arabia 154 : 0.74 |0.28, 1,76) V7<br>Djibouti Wm . 0.61 [-1.60, 0.33) 0.33) 1.25<br>Tunisia R2 R2 135 H 0.27 (0.53, 002) (0.53, 002) 002) 1a43<br>Jorcian R2 R2 561 - 0:30 (-0.72, 0.12) (-0.72, 0.12) 0.12) 5.89<br>Conga & - 0.44 (1.71, 083) (1.71, 083) 083) 078<br>ania o10K124.145) ar<br>Pooua New Guinea New Guinea Guinea 2 - 0.31 (.0.36,0.97) 278<br>Kosovo 7” 0.15 (-0.48, 0.18) 0.18) 12<br>Serbia 1429 ' 0.21 (0.97, 0.48) 1589<br>North Macedonia Macedonia 464 | —— 0.59 (0.17, 1.00) 7.07<br>Overos, Iv Iv 1976 <p 0.05 (-0.16, 0.06) 0.06) 100.09<br>(= 54.4%, p= 0.012)<br>- o 1<br><!-- End of picture text -->

<!-- Start of picture text --> Age 39- Age 40+<br>No. log Odds Odds ratio. % No. log Odds Odds ratio %<br>Study pts (95% cl) Welght Study pis (95%CH Weight<br>Kuwail 986 0.12 (-0.28, 0.51) 0.51) 383 Kuwait 762 0.44 (0.76, 0.08) 693<br>Jordan RI 2674 0.11 (0.06, 0.28) 20.28 Jorcian RI RI 1756 0.09 (-0.30, 0.12) 0.12) 27.03<br>Soudi Arabia Arabia 283 + 0.20 (-0.95, 0.35) 143 Saudi Arabia Arabia 154 : 0.74 |0.28, 1,76) V7<br>Djibouti 993 i 0.14 [-0.17, 0.46) 0.46) 601 Djibouti Wm . 0.61 [-1.60, 0.33) 0.33) 1.25<br>Tunisia R2 R2 317 ' 0.10(0.27,007) 2088 Tunisia R2 R2 135 H 0.27 (0.53, 002) (0.53, 002) 002) 1a43<br>Jorcian R2 R2 1180 - 0.02 [-0.24, 0.28) 0.28) 877 Jorcian R2 R2 561 - 0:30 (-0.72, 0.12) (-0.72, 0.12) 0.12) 5.89<br>Congo a +__ 0500008, 094) 094) 3.12 Conga & - 0.44 (1.71, 083) (1.71, 083) 083) 078<br>aria ass 0705203 301 ania o10K124.145) ar<br>Papua New Guinea New Guinea Guinea v2 : —_—@£$. 07 (045, 1.10) 573 Pooua New Guinea New Guinea Guinea 2 - 0.31 (.0.36,0.97) 278<br>Kosovo W97 - 0.02 [-0.19, [-0.19, 0.23) 1377 Kosovo 7” 0.15 (-0.48, 0.18) 0.18) 12<br>Sorbla 1518 —— 0.34 (0.07, 0.62) 0.62) 7.80 Serbia 1429 ' 0.21 (0.97, 0.48) 1589<br>North Macedonia Macedonia 751 ~ 0.15 (0.48, 0.19) (0.48, 0.19) $37 North Macedonia Macedonia 464 | —— 0.59 (0.17, 1.00) 7.07<br>Overt, IV IV 15237 > 0.09 (0.01, 0.17) 0.17) 100.00 Overos, Iv Iv 1976 <p 0.05 (-0.16, 0.06) 0.06) 100.09<br>(= 66.6%, p < 0.001) (= 54.4%, p= 0.012)<br>| 0 1 - o 1<br>Male Female<br>study ptsNo,No, log(95%Odds Cl)(95%Odds Cl)Odds Cl) Cl) ratio Welght®® study No.ptspts log(958cH) Odds ratio(958cH) Odds ratio Odds ratio ratio Weight%%<br>Kuwait 1162 — 0.09 [-0.30, 0.48) 418 Kuwait 613 - 0.17 (-0.29,0.64) 2.75<br>Jorden RI RI 1a72 0.06 [-0.15, 0.27) 1425 Jordan RI RI 2536 0.32 (0.14,0.50) (0.14,0.50) 18.36<br>Saud Arabia 216 0.04 (0.87, 0.79) 092 Saudi Arabic 213 : 0.59 (-1.29,0.11) 1.21<br>Djibouti 637 1 0.03 (-0.42, 0.37) (-0.42, 0.37) 0.37) au Djibouti 495 4 0.36 (097,079) (097,079) 3.16<br>Jordan R2 R2 su _3—_ 0.48 (0.07, (0.07, 0.88) al Jordan RZ RZ 1224 i 0.23 (-0.03, 0.48) (-0.03, 0.48) 0.48) 897<br>Lobanon R2 1824 0.12 (0.14, 037) 949 Lebanon R2 R2 2399 - 0.11 (0.12033) 11.62<br>Zambia 1802 — 0.40 (0.20, (0.20, 0.59) 16.67 Zombia me a 0,55 (0.31,0.80) (0.31,0.80) 10.07<br>Kosovo: 1253 0.14 [-0.12, 0.40) 916 Kosovo: 1322 0.34 (0.10, 0.59) 9.88<br>Serbia vai Hi 0.16 (0.16, 0.47) (0.16, 0.47) 646 Serbia 1501 -. 0.45 (0.19, 0.70} (0.19, 0.70} 0.70} 9.07<br>North Macedonia: Macedonia: 619 he 0.20 [-0.16, [-0.16, 0.56) Av? North Macedonia Macedonia 543 0.33 (-0.08, (-0.08, 0.74) 3.58<br>Ukraine 316 0.15 (0.55, 085) 1.28 Ukraine 383 ~ 0.47 (-0.04,0.98) (-0.04,0.98) 2.26<br>BeleHaitiHaiti 41771515 FE-- 0.26-0.33 (0.09,[-1.70,0.43)1.04)-0.33 (0.09,[-1.70,0.43)1.04) (0.09,[-1.70,0.43)1.04)[-1.70,0.43)1.04)0.43)1.04)1.04) 21.34omom BelizeHaitiHaiti 2701135135 ~ 0.37 (0.15,0.59)-0.02-0.02 {-1.13, 1.09} 12.370.480.48<br>Jamaica $72 —+—| : 0.49 (0.93, -0.05) 3.23 Jamaica 996 : 0.10 (041,021) (041,021) 6.22<br>Overall IV IV 16490 © 0.18 (0.10,0.26) (0.10,0.26) 100.00 Overall, IV 16179 > 0.29 (0.22,0.37) 100.00<br>(= 36.1%, 36.1%, p =0.087) (= 42.8%,p = 42.8%,p =p = = 0.045)<br>1 o I a ° 1<br><!-- End of picture text -->

<!-- Start of picture text --> Male<br>study ptsNo,No, log(95%Odds Cl)(95%Odds Cl)Odds Cl) Cl) ratio Welght®®<br>Kuwait 1162 — 0.09 [-0.30, 0.48) 418<br>Jorden RI RI 1a72 0.06 [-0.15, 0.27) 1425<br>Saud Arabia 216 0.04 (0.87, 0.79) 092<br>Djibouti 637 1 0.03 (-0.42, 0.37) (-0.42, 0.37) 0.37) au<br>Jordan R2 R2 su _3—_ 0.48 (0.07, (0.07, 0.88) al<br>Lobanon R2 1824 0.12 (0.14, 037) 949<br>Zambia 1802 — 0.40 (0.20, (0.20, 0.59) 16.67<br>Kosovo: 1253 0.14 [-0.12, 0.40) 916<br>Serbia vai Hi 0.16 (0.16, 0.47) (0.16, 0.47) 646<br>North Macedonia: Macedonia: 619 he 0.20 [-0.16, [-0.16, 0.56) Av?<br>Ukraine 316 0.15 (0.55, 085) 1.28<br>BeleHaitiHaiti 41771515 FE-- 0.26-0.33 (0.09,[-1.70,0.43)1.04)-0.33 (0.09,[-1.70,0.43)1.04) (0.09,[-1.70,0.43)1.04)[-1.70,0.43)1.04)0.43)1.04)1.04) 21.34omom<br>Jamaica $72 —+—| : 0.49 (0.93, -0.05) 3.23<br>Overall IV IV 16490 © 0.18 (0.10,0.26) (0.10,0.26) 100.00<br>(= 36.1%, 36.1%, p =0.087)<br>1 o I<br><!-- End of picture text -->

<!-- Start of picture text --> Female<br>study No.ptspts log(958cH) Odds ratio(958cH) Odds ratio Odds ratio ratio Weight%%<br>Kuwait 613 - 0.17 (-0.29,0.64) 2.75<br>Jordan RI RI 2536 0.32 (0.14,0.50) (0.14,0.50) 18.36<br>Saudi Arabic 213 : 0.59 (-1.29,0.11) 1.21<br>Djibouti 495 4 0.36 (097,079) (097,079) 3.16<br>Jordan RZ RZ 1224 i 0.23 (-0.03, 0.48) (-0.03, 0.48) 0.48) 897<br>Lebanon R2 R2 2399 - 0.11 (0.12033) 11.62<br>Zombia me a 0,55 (0.31,0.80) (0.31,0.80) 10.07<br>Kosovo: 1322 0.34 (0.10, 0.59) 9.88<br>Serbia 1501 -. 0.45 (0.19, 0.70} (0.19, 0.70} 0.70} 9.07<br>North Macedonia Macedonia 543 0.33 (-0.08, (-0.08, 0.74) 3.58<br>Ukraine 383 ~ 0.47 (-0.04,0.98) (-0.04,0.98) 2.26<br>BelizeHaitiHaiti 2701135135 ~ 0.37 (0.15,0.59)-0.02-0.02 {-1.13, 1.09} 12.370.480.48<br>Jamaica 996 : 0.10 (041,021) (041,021) 6.22<br>Overall, IV 16179 > 0.29 (0.22,0.37) 100.00<br>(= 42.8%,p = 42.8%,p =p = = 0.045)<br>a ° 1<br><!-- End of picture text -->
# **4. Discussion**

## **4.1. Findings Summary**

To identify the effect of behaviorally informed vaccine messaging on vaccine intentions, we conducted 28 randomized survey experiments about COVID-19 vaccination across 23 mostly low- or middle-income countries and territories, recruiting participants through social media. In this individual participant data meta-analysis, among the unvaccinated, we found that behaviorally informed messages significantly increased the odds of having the intention to get vaccinated by 1.28 times overall, with significant impacts of specific framings ranging from 1.16 times (variant framing) to 1.45 times (experts and religious leaders framing), compared to the control group (i.e., no message). The messages using experts and religious leaders showed the largest impacts, followed by endemic, messages with experts and celebrities, efficacy and pro-social, messages with experts, variants, and dynamic social norms. However, the safety message did not significantly increase vaccination intentions across studies. While some framings exhibited larger effects than others, it is important to interpret these impacts considering differences in implementation—as some framings (e.g., experts and safety) were used across more than a dozen studies while other framings were tested in fewer studies as the pandemic evolved (e.g., variants). In addition, it should be reiterated that our findings represent effects for populations who are active on social media and self-selected into a survey about COVID-19, and outcomes reflected selfreported intentions immediately following message exposure.

Messages using experts and celebrities, messages using experts and religious leaders, and dynamic social norms messages had consistent impacts across the countries and studies, indicated by non-significant heterogeneity in effect sizes. Conversely, the endemic, safety, experts, efficacy and prosocial, and variant messages showed moderate-to-high between-study heterogeneity in effect sizes, with impacts varying across studies and countries. Still, none of our tested study-level moderators, including country-level vaccination rates, intervention start dates, and GDP per capita PPP, significantly explained the heterogeneity. Marginally, safety messages have larger impacts in countries with higher vaccination rate, and expert messages have larger impacts when the intervention was conducted later. Lastly, we found most messages have consistent impacts across observable individual demographics, given the minimal individual-level heterogeneity by demographic characteristics, except larger impacts of expert and safety messages for younger people as well as efficacy and pro-social messages for women.

## **4.2. Implications for Behaviorally-Informed Communication in Practice**

When addressing hesitancy, messages focusing on the underlying concerns and drivers of vaccine acceptance can increase intentions to vaccinate. The observed heterogeneity in the effects of specific message framings suggests that the effectiveness of messages may vary based on contextual factors. Some messages, such as those involving experts and celebrities or religious leaders, showed consistent impacts across studies and countries, while others, such as safety messages, were not effective consistently across countries.

That some message impacts varied by context highlights the importance of testing messages with the target population before implementing communications widely. This also underlines the importance of developing messages tailored to specific contexts and populations to maximize their 24
effectiveness, aligning with prior research emphasizing the importance of message customization. This study also demonstrates the potential for public health authorities to use rapid online survey experiments to promptly identify vaccine hesitancy drivers (e.g., Badiani-Magnusson et al., 2022; Hoy et al., 2022, 2023; Lim et al., 2024; Margolies et al., 2023) and test messages in fast-evolving environments like the pandemic among a growing population within LMICs—social media users. There are over 5 billion smartphone users and over 3 billion Facebook daily users globally (Statista, 2023), underscoring the relevance of these findings despite our study samples not being nationally representative on dimensions like education. Furthermore, as we observe advances in the ability to deliver tailored communication to individuals through technologies like online social media, future research can also examine if and how messages tailored to personalized concerns and needs can increase vaccination intentions.

Our results highlight the importance of identifying and working with multiple trustworthy messengers, such as healthcare professionals, celebrities, and religious leaders, to convey messages effectively, given the consistently significant message impacts using experts and celebrities or religious leaders across countries. This aligns with prior research emphasizing the role of messengers in influencing attitudes and behaviors (Alatas et al., 2024; Huang & Sundar, 2022; Lu et al., 2021; Ruggeri et al., 2024). Interestingly, our results show that using multiple trusted messengers can bolster message effectiveness, while using experts (e.g., “national and international experts”) alone did not result in the consistent impacts on vaccination intentions across countries. Public health campaigns should consider partnering with these trusted and popular figures to amplify their messaging and increase its impact while future research can examine additional effects of consensus among multiple information sources.

Given the consistently significant impacts of dynamic social norms messages across countries, public health authorities in low- and middle-income countries could benefit from leveraging these social norms messages. Consistent with this, previous studies have shown that social norms messages focusing on descriptive norms (e.g., widespread vaccine acceptance) can effectively increase vaccination intentions (Ryoo & Kim, 2021; Sinclair & Agerström, 2021). Future research could explore strategies for creating injunctive social norms (Martinez et al., 2023) to complement dynamic norms in contexts where norms are not yet prominent to identify the best methods of promoting vaccination without triggering unintended consequences such as boomerang effects.

Our study also adds to the literature on message fatigue and health risk communication in dynamically evolving situations in that using diverse messages that shift contexts, such as endemic and variant framings, may help mitigate the fatigue and keep messaging relevant. These messages acknowledging the ongoing risks of COVID and highlighting that vaccination is protective even in contexts where disease is endemic were effective in increasing intentions, suggesting that nuanced information reflecting the experience of the public can be effective. This finding underscores the need for ongoing message adaptation and evolution to maintain engagement and effectiveness over time, especially in prolonged public health crises like the COVID-19 pandemic and slow-burning crises.

# **4.3. Study-Level and Individual-Level Moderators**

While the message framing focusing on endemic, safety, experts, efficacy, and prosocial, and variants showed moderate-to-high heterogeneity in message impacts across studies and countries, we found that country-level vaccination rates, intervention start dates, and GDP per capita PPP do not 25
significantly explain the varying message impacts across studies and countries. We found only one case where a moderator was significant: the intervention date for the expert messages. The intervention date explained the differences in expert messages’ effects, indicating that expert messages were more effective later than earlier. Marginally, intervention start date explained the differences in endemic framing ( _p_ = 0.068), indicating that endemic messages were more effective earlier than later, and perceived vaccine effectiveness was associated with the impact of the variants framing ( _p_ = 0.092), indicating that variants framings were more effective when people perceived the vaccine’s effectiveness. This may be because the remaining unvaccinated population in these countries might be more concerned about vaccine safety and experts’ guidance as more people in the country become vaccinated, or because the messages resonate with their personal experiences and become more credible seeing others around them getting safely vaccinated and a great number of experts have endorsed the vaccines. This also may be because in countries with high perceived vaccine effectiveness, mentioning emerging variants reminded unvaccinated individuals of vaccines' roles and increased their vaccination intentions.

The relative lack of significant individual-level heterogeneity in our study suggests that these messages can have relatively consistent effects across diverse demographic populations active on social media. Only larger effects of expert and safety messages for people younger than 40 years old (vs. 40 years or older) and larger effects for efficacy and pro-social messages for women were found. The literature points to several plausible explanations. For example, messaging about personal efficacy of vaccines may increase motivation among women who perceive men as particularly at risk of negative complications from COVID-19 (Ackah et al., 2022) while enhanced effectiveness of prosocial messages may reflect higher levels of prosocial orientations among women (Rand et al., 2016). Safety messages might appeal to younger adults maybe because, unlike older adults, they were at relatively lower risk of complications from COVID-19 and might have decided to wait and see due to lower perceived susceptibility and impacts like in some countries (e.g., Hamel et al., 2021). Future studies can further explore why these specific framings had differential impacts on younger (vs. older) adults and women (vs. men).

The finding that many message framings were effective across different demographics is particularly relevant for global health efforts, indicating behaviorally informed communications can be broadly effective on users of social media. However, our findings might also reflect the relative importance of individual characteristics beyond demographics when it comes to the impact of behaviorally informed communications (Christy et al., 2022). Future studies should also examine psychological variables, such as specific vaccine safety concerns, in their individual-level moderator analyses.

Our findings have several implications for policy makers and public health authorities aimed at increasing COVID-19 vaccination uptake. The significant increase in vaccination intentions with behaviorally informed messages suggests that messaging designed with behavioral science principles can be a powerful tool in addressing vaccine hesitancy. By leveraging specific message framings, public health campaigns can tailor their messaging to different segments of the population, addressing varying concerns and motivations.

26
# **4.4. Limitations**

This study has limitations that are important to consider when drawing conclusions from our evidence. Due to its rapid introduction and promotion on a large scale, our findings about COVID-19 vaccine hesitancy might not be applicable to other long-established vaccines (e.g., MMR, DTaP, flu) or new vaccines (HPV). Recruitment and survey administration on Facebook, while innovative for mirroring national demographics using Facebook targeting and quota sampling, cannot provide a representative sample, leading to under-representation of older age groups, the poorest, and least educated, who might be disproportionally impacted by the COVID-19 and future pandemics. Self-selection bias may introduce systematic differences between respondents and non-respondents based on unmeasured factors and affect the findings. As such, the generalizability of the findings from our meta-analysis to offline populations remains to be tested and the impacts of our studies likely represent upper-bound estimates of effects. Despite these limitations, the sampling strategy of stratifying by demographic characteristics allowed for greater representation and timely data collection among key subgroups (e.g., women) who can be harder to reach in online settings.

The study measured self-reported vaccination intentions, not actual behavior. While behavioral intentions are commonly used to predict behaviors in experiments, and one meta-analysis found that a medium-to-large change in behavioral intention ( _d_ = 0.66) led to a small-to-medium change in behaviors ( _d_ = 0.36) (Webb & Sheeran, 2006), the extent to which changes in intentions translate into increased vaccinations across these international contexts remains unknown and warrants further investigation, when feasible (e.g., Alhajji et al., 2023). Indeed, some research has identified differential impacts of messaging interventions on vaccine intentions compared to behavior (Saccardo et al., 2024).

Lastly, we were limited in exploring additional study-level moderators (e.g., World Value Surveys [2024]; Hofstede’s cultural dimensions [Culture Factor Group, 2024]) given scarcity of data available for LMICs. While we were able to draw from the Wellcome Global Monitor Indicators to explore the moderating role of country-level beliefs and trust in healthcare workers and beliefs about vaccines, these concepts may be better explored at the individual level, as evidenced by modest empirical evidence for aggregation to the country level described in Appendix D. Our analysis focused on only demographic characteristics as individual-level moderators, as other potentially relevant factors were omitted from surveys to keep them brief for participants. Future research could explore additional moderating factors that might have more explanatory power.

# **4.5. Conclusions**

This study is part of the World Bank's COVID-19 response, aimed at diagnosing vaccine hesitancy and related factors, and informing and guiding real-time risk communication efforts across multiple governments simultaneously. While the WHO declared the end of the COVID-19 pandemic in May 2023 (WHO, 2023), ensuring high immunization rates remains crucial for public health. Our findings demonstrate that behaviorally-informed messages have the potential to nudge vaccine-hesitant individuals towards vaccination. Impacts of such messages can represent significant public health gain, as up to 6 million people were vaccine hesitant in these 23 countries and territories by the time of the study. However, one lesson from our study is that a "one-size-fits-all" approach will not work. Our research emphasizes the importance of testing and tailoring messages to specific audiences and contexts, as the effectiveness of some messages varied across countries. This study highlights the value 27
of conducting rapid experiments in LMICs to inform their communication and policies to address emerging health challenges.

Public health authorities must adapt and test their communication strategies to evolving situations, following best practices, emphasizing transparency, and helping the public accept uncertainty (Liu et al., 2021). For example, we initially tested the safety messages stating "no reported hospitalizations in {country} due to vaccinations" at the time of study to the best of our knowledge. However, rare severe reactions after COVID-19 vaccination have since been reported, despite the benefits of COVID-19 vaccination continuing to outweigh any potential risks (US CDC, 2023). Real-time research can adapt the messages tested.

To prevent future health emergencies and pandemics, public health authorities must expand vaccination coverage in humans and animals (Rupasinghe et al., 2024). Our findings can guide future global immunization efforts, including new vaccines like malaria and HPV. Looking ahead, leveraging social media for data collection can facilitate timely research and adaptative messaging strategies. The next step is to apply, transfer, replicate, and test findings in various online and offline real-world settings with various target populations (Saccardo et al., 2024). By continuing to refine our understanding and applying effective risk communication, we can build a robust evidence base to prepare for future health emergencies and pandemics (World Bank, 2022), just as our research contributed to the fight against COVID-19, particularly in low- and middle-income countries and to fortifying global health defenses.

28