metadata: authors: 0: author: Dugas,Michelle Anne Matejovits
1: author: Pinzon Hernandez,Daniel Alejandro
2: author: Lim,Jungkyu Rhys
3: author: Vakis,Renos
4: author: Afif,Zeina
5: author: Hasumi,Takahiro
6: author: Elfadel,Diya Nitham Mousa
count: Jordan,World
docna: 0: docna: Find the Fake : Boosting Resistance to
            Health Misinformation in Jordan with a WhatsApp Chatbot Game
docty: Policy Research Working Paper
theme: Social Inclusion,Participation and Civic Engagement,Human Development and Gender,Data Development and Capacity Building,Social Development and Protection,Disease Control,Health System Strengthening,Pandemic Response,Health Service Delivery,Public Sector Management,Data production, accessibility and use,Health Systems and Policies
abstracts: cdata!: The proliferation of misinformation and
            disinformation threatens to erode the credibility of public
            institutions and limit their capacity to implement policies
            that enhance public well-being. While health misinformation
            represents an urgent global challenge, relatively little
            research has examined solutions in low- and middle-income
            countries. This study experimentally tests the impact of a
            novel WhatsApp chatbot game pre-bunking inoculation
            intervention in Jordan to boost capacity to identify common
            misinformation techniques and reduce the likelihood of
            sharing misleading headlines with
            others&#xf02c;effectively
            &#x201c;inoculating&#x201d; them against health
            misinformation. A sample of 2,851 participants was recruited
            online and randomly assigned to five study arms: (1)
            comprehensive game-based inoculation, (2) brief game-based
            inoculation that highlighted examples of only
            misinformation, (3) infographics-based inoculation, (4)
            exposure to placebo infographics unrelated to
            misinformation, and (5) pure control. To evaluate the impact
            of the intervention, the study assesses two main outcomes:
            (1) ability to discern accurately headlines using
            misinformation techniques and headlines that do not use
            misinformation techniques, and (2) discernment in sharing
            the two types of headlines. Compared to the placebo group,
            the comprehensive game significantly improved discernment of
            misinformation and reduced the likelihood of sharing
            misleading headlines. A brief version of the game yielded
            weaker effects on discernment of misinformation, but
            similarly reduced intentions to share misleading headlines.
            In contrast, exposure to infographics teaching similar
            techniques showed no significant impacts on discernment of
            misinformation, and marginal effects on intention to share
            misleading headlines. These findings suggest that games can
            effectively inoculate the public against misinformation in
            the context of a middle-income country in the short term.
            Future research is needed to explore the boundary conditions
            of the findings.
display_title: Find the Fake : Boosting Resistance
            to Health Misinformation in Jordan with a WhatsApp Chatbot Game
listing_relative_url: /research/2024/09/34396878/find-fake-boosting-resistance-health-misinformation-jordan-whatsapp-chatbot-game
topicv3: ICT Literacy m1326546 1536,Immunizations m1327505 1459,Behavior Change in Public Health m1504351 3656
publishtoextweb_dt: 2024-09-30T00:00:00Z
dois: 10.1596/1813-9450-10933
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
no_of_pages: 62
author_keys: 0: author_key: 2125751
1: author_key: 1723058
2: author_key: 2117014
3: author_key: 167914
4: author_key: 167285
5: author_key: 1623272
6: author_key: 2160215
model_extractions: Policy Research Working Paper 10933

# Find the Fake

Boosting Resistance to Health Misinformation in Jordan with a WhatsApp Chatbot Game _Michelle Dugas Daniel Pinzón JungKyu Rhys Lim Renos Vakis Zeina Afif Takahiro Hasumi Diya Elfadel_ Poverty and Equity Global Practice Health, Nutrition and Population Global Practice & Development Impact Group September 2024 A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click **here** for direct access.
Policy Research Working Paper 10933

# **Abstract**

The proliferation of misinformation and disinformation threatens to erode the credibility of public institutions and limit their capacity to implement policies that enhance public well-being. While health misinformation represents an urgent global challenge, relatively little research has examined solutions in low- and middle-income countries. This study experimentally tests the impact of a novel WhatsApp chatbot game pre-bunking inoculation intervention in Jordan to boost capacity to identify common misinformation techniques and reduce the likelihood of sharing misleading headlines with others, effectively “inoculating” them against health misinformation. A sample of 2,851 participants was recruited online and randomly assigned to five study arms: (1) comprehensive game-based inoculation, (2) brief game-based inoculation that highlighted examples of only misinformation, (3) infographics-based inoculation, (4) exposure to placebo infographics unrelated to misinformation, and (5) pure control. To evaluate the impact of the intervention, the study assesses two main outcomes: (1) ability to discern accurately headlines using misinformation techniques and headlines that do not use misinformation techniques, and (2) discernment in sharing the two types of headlines. Compared to the placebo group, the comprehensive game significantly improved discernment of misinformation and reduced the likelihood of sharing misleading headlines. A brief version of the game yielded weaker effects on discernment of misinformation, but similarly reduced intentions to share misleading headlines. In contrast, exposure to infographics teaching similar techniques showed no significant impacts on discernment of misinformation, and marginal effects on intention to share misleading headlines. These findings suggest that games can effectively inoculate the public against misinformation in the context of a middle-income country in the short term. Future research is needed to explore the boundary conditions of the findings.

This paper is a product of the Poverty and Equity Global Practice, the Health, Nutrition and Population Global Practice and the Development Impact Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at mdugas@worldbank.org. A verified reproducibility package for this paper is available at http://reproducibility.worldbank. org, click **here** for direct access.

<!-- Start of picture text --> TRANSPARENT<br>ANALYSIS<br>G<br>R E<br>N<br>S<br>I P<br>Y E<br>K A<br>C A<br>R P<br>LI R<br>O E<br>O C<br>W R<br>H<br>S<br>P<br><!-- End of picture text --> _The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Find the Fake: Boosting Resistance to Health Misinformation in Jordan with a WhatsApp**

# **Chatbot Game**

Michelle Dugas, World Bank, mdugas@worldbank.org, Daniel Pinzón, World Bank, dpinzonhernandez@worldbank.org, JungKyu Rhys Lim, World Bank, rhyslim@worldbank.org Renos Vakis, World Bank, rvakis@worldbank.org Zeina Afif, World Bank, zafif@worldbankgroup.org Takahiro Hasumi, World Bank, thasumi@worldbank.org Diya Elfadel, World Bank, delfadel@worldbank.org Keywords: Misinformation, Pre-bunking, Information Literacy, Jordan, RCT, Chatbot JEL classification: D.90 Micro-based behavioral-economics, General
# **Acknowledgments**

From the World Bank Team, the study was led by Michelle Dugas, Daniel Alejandro Pinzón Hernández, JungKyu Rhys Lim, Zeina Afif, Renos Vakis at the Mind, Behavior & Development (eMBeD) Unit in the World Bank’s Poverty and Equity Global Practice and Takahiro Hasumi and Diya Elfadel at the World Bank’s Health, Nutrition and Population Global Practice. Additional contributing team members include Ellen Moscoe, Mohamad Chatila, Jana Kontar, and Haena Kim. The authors express their gratitude to all who contributed to the implementation of this project, especially country government partners. The team is grateful to Victor Orozco (Senior Economist, DIME2), Erwin van Knippenberg (Economist, EMNPV), and Samuel Fraiberger (Data Scientist, DIME0), for their valuable comments as peer reviewers. This study was partially funded by the Advancing Health Online (AHO) Initiative, a fiscally sponsored project of Global Impact and funded by Meta and MSD to advance public understanding of how social media and behavioral sciences can be leveraged to improve the health of communities around the world. The findings, conclusions, and recommendations expressed in this article are those of the authors and do not necessarily reflect the views of the collaborators.

2
# **Find the Fake: Boosting Resistance to Misinformation in Jordan with a WhatsApp Chatbot**

# **Game**

With the increasing use of social media, misinformation has proliferated and traveled farther, wider, and faster than ever before, impacting wide-ranging topics including climate change, politics, and vaccines (Vosoughi et al., 2018). Accordingly, misinformation and disinformation are considered among the 32 most severe global risks<sup>1</sup> (World Economic Forum [WEF], 2023), with direct costs estimated at about US$ 78 billion across different countries and sectors (Cavazos, 2019). During the coronavirus disease (COVID-19) pandemic, misinformation emerged as a particularly salient threat to public health given its association with vaccine hesitancy (Loomba et al., 2021; Roozenbeek et al., 2020; Zhao et al., 2023).

To tackle widespread misinformation, researchers have developed several psychological solutions: fact-checks or corrections of specific misinformation claims (i.e., debunking), priming to be cautious of accuracy (i.e., accuracy primes or nudges), and psychological inoculation against misinformation techniques (i.e., prebunking) (American Psychological Association, 2023 [APA]; Lim et al., 2022; Roozenbeek & van der Linden, 2022; Zhao et al., 2023). Evidence suggests that all these solutions can be effective at reducing the spread of misinformation (Lim et al., 2022; Pennycook et al., 2023; Zhao et al., 2023), but teaching common misinformation techniques as a form of inoculation holds the advantage of being both proactive and effective in combatting misinformation. In contrast, although debunking showed better non-misinformation credibility assessment and sharing intentions than prebunking in some research (Lu et al., 2023), debunking requires significant resources to detect and respond to specific claims and there

> 1 According to World Economic Forum (2023), “Global risk” is defined as the possibility of the occurrence of an event or condition which, if it occurs, would negatively impact a significant proportion of global GDP, population or natural resources.

3
remains significant debate about potential backfire effects of making misinformation claims more salient and memorable (APA, 2023; Ecker et al., 2022; Nan et al., 2022). In addition, effectiveness of accuracy primes may be limited by challenges in reaching all platforms where people are exposed to misinformation at scale (Roozenbeek & van der Linden, 2022; Traberg et al., 2022; van Bavel et al., 2021).

While there is a growing body of evidence supporting the effectiveness of inoculation against misinformation, this research remains largely limited to Western, educated, industrialized, rich, and democratic (WEIRD) societies (Henrich et al., 2010; Nan et al., 2022) with a few notable exceptions (e.g., Arechar et al., 2023; Athey et al., 2022; Bilo-Thomas et al., 2021; Harjani et al., 2023). However, many threats from misinformation are found in other regions, including the Middle East. For example, a survey of Jordanian Arab residents found that over 60% believed that COVID-19 is a man-made virus, and about 25% believed that the COVID-19 vaccine is a way to implant microchips into people and that vaccines cause infertility, exacerbating vaccine hesitancy (Sallam et al., 2020, 2021). Such misinformation has been circulated extensively through social media (Al-Jalabneh & Safori, 2020; Alkhwaldeh & Emam, 2020; Habes et al., 2023), yet solutions for this cultural context are underexplored.

To address this knowledge gap, we test the effectiveness of a chatbot-based inoculation game with comprehensive and brief versions (i.e., active inoculation), infographics (i.e., passive inoculation), and a placebo compared to no inoculation through a randomized experiment delivered by WhatsApp in Jordan ( _N_ = 2,851). Specifically, participants learn common misinformation techniques through chatbot-based games or infographics. Then, we measure whether participants can accurately discern headlines using misinformation tactics and their 4
likelihood of sharing misinformation. By doing so, we reveal which inoculation interventions could be more effective to tackle misinformation in this unique context.

# **Literature Review**

# **_How Misinformation Can Be Stopped from Spreading_**

Like vaccination against a virus, inoculation theory poses that people can be psychologically “vaccinated” against misinformation by providing people with tools to counter persuasive attacks (McGuire, 1964). Inoculation can be achieved in two ways: (i) narrowspectrum inoculation based on specific arguments and issues prior to exposure of a persuasive claim, or (ii) broad-spectrum inoculation against commonly used misinformation techniques (e.g., extreme emotions, fake experts) rather than specific claims (van der Linden et al., 2021; Basol et al., 2021). In both cases, the inoculation approach has two components: a pre-warning of the persuasive claim and a pre-emptive refutation (Compton, 2012). Such refutational preemptions help people build “cognitive antibodies” against persuasive arguments (Banas & Rains, 2010). These components, a pre-warning and a pre-emptive argument refutation, contribute to motivational readiness to defend oneself from manipulation and prepare individuals with arguments against misinformation that they might encounter.

A meta-analysis of earlier evidence found that inoculation interventions against persuasion showed a medium effect size ( _d_ = 0.43) on average with decaying resistance after two weeks (Banas & Rains, 2010). Also, a recent meta-analysis also found that inoculation interventions improved credibility assessment for misinformation ( _d_ = -0.36) and nonmisinformation ( _d_ = 0.20) and increased intentions to share non-misinformation ( _d_ = 0.18), yet did not significantly decrease misinformation sharing intentions ( _d_ = -0.35, _p_ = .12) (Lu et al., 5
2023). Consistent with the notion of broad-spectrum inoculation, the meta-analysis also found that inoculation treatments can create resistance not only to the target persuasive message included in the inoculation treatment, but also novel persuasive messages, providing a “blanket of protection” (Banas & Rains, 2010; Compton & Pfau, 2005, p. 105).

Recent research has also supported the efficacy of inoculation against various types of health misinformation (Compton et al., 2016; Isles et al., 2021). In the context of vaccination specifically, prior research found that inoculations before exposure to misinformation were more effective at increasing vaccination intentions than debunking (Jolley & Douglas, 2017). This is consistent with evidence that, once exposed, it is difficult to remove misinformation’s impact; people may continue to be influenced by misinformation or seek misinformation aligned with their social identities (van Bavel et al., 2021). Together, these findings support the use of inoculation to equip the public with the knowledge and skills needed to recognize and stop the spread of misinformation.

# **_Behavioral Inoculation Approaches to Misinformation_**

The inoculation approach can use various modalities, such as text, infographics (i.e., passive inoculation), videos, and games (i.e., active inoculations), with varying degrees of activeness and passiveness (Roozenbeek & van der Linden, 2022; Lewandowsky & van der Linden, 2021). Passive and active inoculations have distinct tradeoffs for potential impact on reducing susceptibility to misinformation. For instance, passive inoculations might be easy to manage and distribute widely (Roozenbeek et al., 2022), yet they may be less engaging, eliciting smaller effect sizes than games (e.g., Basol et al., 2021). Conversely, game-based, or active inoculations, may elicit large effect sizes and their impacts may last longer, but playing games 6
takes more time than reading infographics or watching videos, posing potential challenges for scale (e.g., Maertens et al., 2021).

Some studies found that, across various topics including climate change and health, people better identified misinformation (Basol et al., 2021; Roozenbeek et al., 2020), had more confidence to identify misinformation (Basol et al., 2020, Saleh et al., 2021), and lower willingness to share misinformation after exposure to game-based inoculation than viewing infographics (Basol et al., 2021; Roozenbeek & van der Linden, 2020). People also better identified misinformation after reading infographics compared to no inoculation, yet with slightly lower accuracy compared to those exposed to game-based inoculation (Basol et al., 2021).

Positive impacts for game-based inoculation have been supported in different languages (e.g., French, German; Basol et al., 2021; Roozenbeek et al., 2020) and such impacts have lasted at least about three months (Maertens et al., 2021), though evidence is mixed on how to most effectively implement prebunking (Athey et al., 2022). Given prior research findings that both active game-based and passive inoculations can effectively increase the accuracy of discerning misinformation, with some indication that passive inoculations yield smaller impacts than active inoculations, we pose the following hypotheses:

**H1:** Active inoculations (i.e., game-based) (H1a) and passive inoculations (i.e.,

infographics) (H1b) will increase the accuracy of discerning misinformation, compared to the placebo group.

**H2:** Active inoculations (i.e., game-based) will significantly increase the accuracy of discerning misinformation compared to passive inoculations (i.e., infographics).

7
**H3:** Active inoculations (i.e., game-based) (H3a) and passive inoculations (i.e.,

infographics) (H3b) will increase discernment in sharing information, compared to the placebo group.

**H4:** Active inoculations (i.e., game-based) will increase discernment in sharing information compared to passive inoculations (i.e., infographics).

Prior research found that those who played inoculation games were more willing to share the intervention than those who read related infographics (Basol et al., 2021). Thus, we pose the following hypothesis:

**H5:** Active inoculations (i.e., game-based) will increase the willingness to recommend the chatbot.

Notably, prior research suggested that playing prebunking games does not impact the ability to discern non-misinformation but instead makes participants respond conservatively, more frequently judging any news item as ‘false’, including non-misinformation (Basol et al., 2021; Modirrousta-Galian & Higham, 2023a, 2023b). While increased skepticism may reduce the spread of misinformation, some argue that non-discrimination between credible and noncredible information could be harmful in the long run (Guay et al., 2023).

Given the criticisms that inoculation may increase skepticism of all information rather than improve discernment between true news and misinformation (Modirrousta-Galian & Higham, 2023a), we aim to explore mechanisms that could mitigate backfire effects on credible information. To this end, we test whether comprehensive training that includes examples of information that is misleading and information that is _not_ misleading is critical to improved discernment. We contrast the comprehensive inoculation against a brief active inoculation 8
condition (i.e., short games) that includes only examples of misinformation, an approach similar to early inoculation theory studies (e.g., Banas & Rains, 2010; McGuire & Papageorgis, 1962) that included only attack messages and refutational pre-emptions. Given no prior research, we explore the extent to which the impact of the brief active inoculation (i.e., short game) on our outcomes, and consider how its impact differs from that of the comprehensive active inoculation.

# **Method**

We conducted a randomized experiment using WhatsApp. This study was approved by the Health Media Lab Institutional Review Board (#2118). We closely coordinated with the Ministry of Health, the Hashemite Kingdom of Jordan during the design, implementation, and analyses of the study.

# **Recruitment**

We conducted this study between October 27, 2022, and November 23, 2022. Participants were recruited through Facebook advertisements targeting users aged 18 years or older and located in Jordan with WhatsApp installed. The advertisements (see **Figure** **_1_** ) marketed our chatbot-based game under the title “Find the Fake”, inviting people to play a challenge related to the spread of misinformation online for a chance to win 70 Jordanian dinars (roughly equivalent to US$ 100).

9
demographics. The evaluation and survey were presented directly following the intervention implementation; therefore, outcomes represent measures of immediate term effects.

**Table 1. Summary of Study Arms**

|**Study Arm**|**Description**|
|---|---|
|Comprehensive active inoculation|Completed three lessons on misinformation (appeal to<br>emotions, false experts, digital manipulation of<br>documents/images), including examples of what<br>misinformation looks like and what misinformation<br>does_not_look like. Users received feedback on their<br>performance in the form of scores and badges.|
|Brief active inoculation|Completed three lessons on misinformation (appeal to<br>emotions, false experts, digital manipulation of<br>documents/images), including only examples of<br>misinformation. Users received feedback on their<br>performance in the form of scores and badges.|
|Passive inoculation|Exposed to three infographics about misinformation<br>(appeal to emotions, false experts, and how to spot<br>false content), and asked to complete attention checks.|
|Placebo|Exposed to three infographics on an unrelated topic<br>(road safety) and asked to complete attention checks.|
|Control|No training or interaction before evaluation.|

# **_Comprehensive Active Inoculation_**

Participants assigned to the comprehensive active inoculation arm completed three rounds of training that addressed three common misinformation techniques: use of content design to elicit strong emotions (e.g., use of very negative words like ‘horrible’), reference to false experts (e.g., appealing to authorities who are expert in a domain unrelated to the issue at hand), and the digital manipulation of documents/images with fake content.

In each round, participants were presented with different vignette-based scenarios reflecting domains where misinformation is commonly spread including political and health issues. After a brief description of the scenario, participants were presented with messages 11
shared by different characters and asked to indicate whether they believed the messages were spreading misinformation or not. In the comprehensive version of the game, messages included examples using misinformation spreading techniques and examples of messages that did _not_ use misinformation techniques.

After each question, participants would get feedback about whether their responses were correct or incorrect, and the chatbot leveraged several gamification strategies. For example, participants received visual feedback about their scores that persisted throughout each round (e.g., stars for correct responses and blank squares for each incorrect response). In addition, participants who correctly identified the message spreading misinformation in a round received a virtual ‘badge’. At the end of the game, participants received a recap of their performance and the various badges they earned. See **Appendix** **_A_** for screenshots from the game.

# **_Brief Active Inoculation_**

The brief active inoculation condition was the same as the comprehensive inoculation game except participants were exposed to only examples of messages that used misinformation techniques. To shorten the game, all neutral examples were excluded from the training rounds. **_Passive Inoculation_** Participants assigned to passive inoculation were sent three infographics adapted from the UNESCO resources on media and information literacy (UNESCO, 2022) with imagery and text referencing COVID-19 removed. The three infographics (see **Appendix** **_B_** ) covered information that paralleled the lessons taught in the three rounds of gamified training. Specifically, they addressed: (1) that disinformation was often designed to trigger emotional reactions, (2) that false experts are often used to spread manipulative information, and (3) how to spot false content and rumors. The infographics were described to users as tips that would help them perform well 12
on the ‘game’ that began after they learned the tips, which actually consisted of our evaluation questions. After each infographic, participants were asked to answer an attention check question (e.g., ‘Please reply with “3”. Do not reply with other options.’). The attention check served as a placebo for interaction similar to the game experience, but without actively testing users’ understanding of the misinformation lesson.

# **_Placebo_**

The procedure of the placebo arm followed the same protocol as the infographics study arm except all infographics were unrelated to misinformation. The featured infographics were published by the World Health Organization (WHO, 2022) and instead focused on three topics of road safety: (1) traffic injury facts; (2) speed management; (3) how to keep children safe (see **Appendix** **_C_** for placebo infographics). This activity was framed as a practice round for users in which they could learn the game mechanics of how to respond to questions, with the ‘real’ game (i.e., evaluation) beginning after the practice.

# **_Control_**

In the control condition, there was no exposure to any material, and participants proceeded directly to the outcome evaluation. The outcome evaluation was framed as the game for participants in this study arm.

# **_Measures_**

**Evaluation Outcomes.** To evaluate the impact of the misinformation treatments, the

primary outcomes of interest were: (i) rates of accurately discerning between headlines that use misinformation tactics and those that do not, and (ii) discernment in sharing headlines (see **Appendix** **_D_** for evaluation measures). These outcomes were assessed immediately after the implementation of the intervention.

13
**_Misleading Headlines._** All headlines focused on COVID-19 as a theme, and we

generated new headlines based on existing headlines rather than using headlines from published articles. This approach was adopted to ensure that all headlines were completely novel to participants whereas participants could vary in their familiarity with real-world headlines.

The designed headlines were adapted from prior research testing the effectiveness of inoculation against COVID-19 misinformation (Basol et al., 2021), and reflect the three common tactics used in misinformation that were also the focus of the inoculation training (i.e., emotional appeals, false experts, and fake documents/images). Additionally, the misleading headlines were designed to address themes similar to misinformation that had spread online according to various fact-checking sources (e.g., AFP Fact Check, <u>Africa Check) to enhance ecological validity.</u> **_Judgements of Misinformation._** Adapted from Roozenbeek et al. (2022), participants were presented with six headlines: three using misinformation tactics and three that did not use common misinformation tactics. When presented with each headline, participants were asked to respond to the question ‘ _Does this headline use any misinformation techniques_ ?’ on a 4-point scale: Definitely is misinformation, Probably is misinformation, Probably is not misinformation, Definitely is not misinformation. For ease of interpretation, ratings were scored such that higher scores represent stronger belief that a headline was misinformation ( _Definitely is misinformation_ = 4 and _Definitely is not misinformation_ = 1). We chose phrasing that highlighted the term misinformation to increase relevance to policymakers and practitioners, but it is worth noting that much prior research on inoculation measures perceptions of misleadingness rather than misinformation classifications per se (e.g., Roozenbeek et al., 2022).

In line with prior literature, we compute three scores to assess accuracy in misinformation detection (Basol et al., 2021; Maertens et al., 2021). First, we calculate a measure of 14
_discernment_ , defined as a participant’s average misinformation scores for misleading headlines minus their average score for headlines without misleading content. With this operationalization, discernment scores could range from -3 to +3 where a score of +3 indicates a participant rated all misinformation headlines as ‘Definitely misinformation’ and all non-misinformation headlines as ‘Definitely not misinformation’ for perfect discernment.

We also examine the disaggregated discernment score including the average ratings for the three misleading headlines and ratings for the three non-misleading headlines. As higher ratings correspond to judgements that a headline is using misinformation tactics, more accurate scores would be represented by higher scores on the misleading headlines (representing true positives) and lower scores on the non-misleading headlines (representing true negatives).

**_Sharing Misinformation_** _._ Adapted from Basol et al. (2021) and Roozenbeek and van der Linden (2020), sharing of misinformation was assessed with two headlines: one that did not use misinformation tactics and a headline that used misinformation tactics. Each participant was randomly assigned to one of two misleading headlines, a headline that used extreme emotion or a headline that used a false expert. Participants were asked to rate their likelihood of sharing each of the headlines on a four-point scale: (1) Very unlikely to share, (2) Unlikely to share, (3) Likely to share, (4) Very likely to share.

As with judgements of misinformation, we report three scores for sharing—discernment of sharing, likelihood of sharing misleading headlines, and likelihood of sharing non-misleading headlines. Discernment of sharing was calculated as the sharing score for non-misleading headline minus the sharing score for the misleading headline. Accordingly, participants with high positive discernment of sharing have _lower_ intentions of sharing misinformation, relative to sharing more non-misinformation.

15
# **Self-Report Outcomes.** Complementing the evaluation of the intervention’s impact on

detection and sharing of misinformation, we examined differences in attitudes toward the chatbot game.

**_Confidence_ .** Participants were asked to indicate the extent to which they felt more, the same, or less confident in detecting misinformation after completing the game.

**_Perceived Difficulty_ .** Participants were asked to report whether they thought the game was too difficult, the right level of difficulty, or too easy.

**_Recommending the Game_ .** Participants were asked to indicate the extent to which they would recommend the game to others with three possible response categories: yes, maybe, and no.

# **_Participants_**

A total of 2,851 participants completed the study. Of them, 63% identified as male and 33% as female; 49% reported having completed secondary education (3% with no education, 12% with primary, and 3% with tertiary education); 53% reported being between the ages of 18 and 29, 25% in their 30s, 13% in their 40s, and 5% over the age of 50. Finally, 85% reported being vaccinated for COVID-19, 5% unvaccinated but willing to vaccinate, and 5% unvaccinated and unwilling to vaccinate. None of the treatment arms reported significant differences in demographics and vaccination status compared to the control group. See **Appendix** **_E_** for sample composition and randomization balance and **Appendix** **_F_** for age and gender distributions of users who started WhatsApp conversations according to Facebook Ad Manager data.

# **Results**

# **Attrition and Study Completion Time**

16
<!-- Start of picture text --> Facebook Ads<br>recruitment<br>(n=6,303)<br>Intro and opt-in<br>(n-4,150)<br>Comprehensive Active Brief Active Passi Control<br>Inoculation(n=879) Inoculation(n=841) assive(n=896)Inoculation - No(n=586)Trainingra -<br>Intervention complete<br>(n=3,407) n=672 (76%) n=717 (85%) n=718 (80%) n=714 (75%) n=586 (100%)<br>82% |<br>Evaluation<br>(n=2,887) n=550 (63%) n=581 (69%) n=625 (70%) n=614 (65%) n=517 (88%)<br>70% |<br>Survey<br>(n=2,851) n=548 (62%) n=574 (68%) n=618 (69%) n=603 (64%) n=508 (87%)<br>69%<br><!-- End of picture text -->
**Table 2.** Results of attrition analysis

||(1)<br>Completed<br>treatment|(2)<br>Started<br>evaluation|(3)<br>Completed<br>evaluation|(4)<br>Started<br>survey|(5)<br>Completed<br>survey<br>(end)|
|---|---|---|---|---|---|
|Control|0.293***|0.274***|0.235***|0.240***|0.231***|
||(0.015)|(0.016)|(0.020)|(0.021)|(0.021)|
|Comprehensive Active|-0.045**|-0.039*|-0.022|-0.016|-0.013|
|Inoculation|(0.022)|(0.022)|(0.023)|(0.023)|(0.023)|
|Brief Active Inoculation|0.027|0.032|0.043*|0.044**|0.046**|
||(0.021)|(0.021)|(0.022)|(0.022)|(0.022)|
|Passive Inoculation|0.068***|0.072***|0.050**|0.054**|0.054**|
||(0.020)|(0.021)|(0.022)|(0.022)|(0.022)|
|Controls|No|No|No|No|No|
|Observations|4,150|4,150|4,150|4,150|4,150|
|R-squared|0.060|0.050|0.031|0.031|0.028|

_Note._ OLS estimates. Base comparison group is the placebo condition. Robust standard errors are shown in parenthesis. _*** p<_ 0.01 _, ** p<_ 0.05 _, * p<_ 0.1.

# **Discernment in Misinformation Judgements**

Before running analyses to test our hypotheses, we summarize the descriptive statistics for our three scores of misinformation judgements (see **Table 3** ). As expected, misleading headlines tended to be perceived as using misinformation tactics to a greater extent than non-misleading headlines. Violin plots depicting the full distribution of scores are also reported in **Appendix H.**

**Table 3.** Descriptive statistics of misinformation discernment scores by study arms

||Misinfo<br>Discer|rmation<br>nment|Misinfo<br>Judgem<br>Misle<br>Head|rmation<br>ents of<br>ading<br>lines|Misinfo<br>Judgem<br>Non-Mi<br>Head|rmation<br>ents of<br>sleading<br>lines|
|---|---|---|---|---|---|---|
||_M_|_SD_|_M_|_SD_|_M_|_SD_|
|Placebo|0.89|1.06|3.08|0.72|2.19|0.86|
|Control|0.96|1.06|3.11|0.72|2.15|0.85|
|Comprehensive Active Inoculation|1.25|1.03|3.22|0.69|1.98|0.84|
|Brief Active Inoculation|1.09|1.01|3.34|0.72|2.24|0.92|
|Passive Inoculation|0.90|1.05|3.14|0.69|2.23|0.85|

18
# **_Effect of Active and Passive Inoculations on Misinformation Discernment (H1, H2)_**

H1 predicted that active inoculations (i.e., game-based) (H1a) and passive inoculations (i.e., infographics) (H1b) will increase the accuracy of discerning misinformation, compared to the placebo group. H2 predicted that active inoculations (i.e., game-based) will significantly increase the accuracy of discerning misinformation compared to passive inoculations (i.e., infographics). To test H1 and H2, we conducted Ordinary Least Squares (OLS) regressions on the three scores to assess misinformation discernment across study arms, controlling for observable demographic characteristics (see **Table** **_4_** ). To address the possibility that misinformation discernment results are due to the experience of the chatbot, differential attrition, or some other external factor, we use the placebo group (instead of the control group) as the comparison group as it followed the same protocol as the infographic group but was not exposed to the misinformation mechanism and learning process. This placebo group mimics the experience of the treatment arms by showing information on road safety and asking follow-up questions to ensure that respondents could learn game mechanism.

# **Impact of active inoculation on misinformation discernment compared to placebo**

**(H1a).** Consistent with H1a, we find that participants assigned both comprehensive and brief active inoculation groups are better at discerning between misinformation and non-

misinformation than the placebo group. More specifically, the comprehensive active inoculation arm increased the accuracy of discerning misinformation by 0.29 _SD_ compared to the placebo group _._ Disaggregating this score, we find misinformation ratings increased 0.16 _SD_ for misleading headlines and decreased 0.22 _SD_ for non-misleading headlines compared to the placebo group, suggesting increased accuracy in classifying both types of information.

The brief active inoculation group that completed the short version of the game scored 0.14 _SD_ higher than the placebo group on misinformation discernment (see **Table 4** and **Figure 3,** Panel 19
A). However, disaggregating the discernment score, we find this effect is driven by improvements in accurately judging misleading headlines as using misinformation tactics, but there is no improvement in judgements of non-misleading headlines. Specifically, the brief active inoculation, containing only examples of misinformation, increased the probability of detecting misinformation by 0.31 _SD_ but did not decrease misinformation ratings of non-misleading headlines (see **Table 4** and **Figure 3** , Panel B).

# **Impact of passive inoculation on misinformation discernment compared to placebo**

**(H1b).** Contrary to H1b, we find that the passive inoculation did not significantly improve discernment (see **Table 4** , Panel A and **Figure 3** ). Disaggregating the discernment score, there is no difference in ratings of misinformation for either misleading or non-misleading headlines compared to the placebo group.

# **Impact of active inoculation on misinformation discernment compared to passive**

**inoculation (H2).** Consistent with H2, the active inoculation groups are better at misinformation discernment than the passive inoculation group (See **Table 4** , Panel B). More specifically, participants of the comprehensive version of the game score 0.33 _SD_ higher on misinformation discernment, and participants of the brief version of the game score 0.18 _SD_ higher, than the passive inoculation infographic group.

20
**Table 4. Regression results for misinformation discernment scores.**

||(1)<br>Misinformation<br>Discernment|(2)<br>Misinformation<br>Judgements of<br>Misleading<br>Headline|(3)<br>Misinformation<br>Judgements of Non-<br>Misleading<br>Headline|
|---|---|---|---|
|**_Panel A: Placebo as base group_**||||
|Control|-0.067|-0.054|0.037|
||(0.059)|(0.061)|(0.058)|
||[0.828]|[0.828]|[0.828]|
|Comprehensive Active|0.285***|0.155***|-0.218***|
|Inoculation|(0.057)|(0.058)|(0.056)|
||[0.000]|[0.026]|[0.000]|
|Brief Active Inoculation|0.140**|0.311***|0.084|
||(0.056)|(0.059)|(0.059)|
||[0.035]|[0.000]|[0.258]|
|Passive Inoculation|-0.041|0.037|0.080|
||(0.056)|(0.057)|(0.054)|
||[0.975]|[0.975]|[0.607]|
|Controls|Yes|Yes|Yes|
|Observations|2,851|2,851|2,851|
|R-squared|0.090|0.040|0.069|
|**_Panel B: Passive Inoculation as base group_**||||
|Control|-0.027|-0.091|-0.042|
||(0.058)|(0.060)|(0.057)|
||[0.909]|[0.563]|[0.909]|
|Placebo|0.041|-0.037|-0.080|
||(0.056)|(0.057)|(0.054)|
||[0.975]|[0.975]|[0.607]|
|Comprehensive Active|0.325***|0.119**|-0.297***|
|Inoculation|(0.056)|(0.056)|(0.055)|
||[0.000]|[0.098]|[0.000]|
|Brief Active Inoculation|0.181***|0.274***|0.005|
||(0.055)|(0.057)|(0.058)|
||[0.006]|[0.000]|[0.938]|
|Controls|Yes|Yes|Yes|
|Observations|2,851|2,851|2,851|
|R-squared|0.090|0.040|0.069|

_Note._ OLS estimates of treatment effects. Control variables are age, education, gender, and vaccination status for COVID-19. Robust standard errors are shown in parenthesis, and Westfall-Young stepdown adjusted p-values in brackets (1,000 bootstrap replications). _*** p<_ 0.01 _, ** p<_ 0.05 _, * p<_ 0.1.

21
<!-- Start of picture text --> N<br>r=<br>ety<br>E<br>E<br>&<br>@<br>a<br>57<br>E<br>=<br>B<br>=<br>oa<br>Control Placebo Camprehensive Brief Passive<br>Active Active Inoculation<br>Inoculatian Inoculation<br><!-- End of picture text --> <!-- Start of picture text --> ap<br>£<br>Fs<br>5,<br>a=]<br>=<br>i=<br>So<br>3<br>E TM<br>iB<br>=<br>Control Placebo Comprehensive Brief Passive<br>Active Active Inoculation<br>Inoculation Inoculation<br>[Es Misleading Headline = [ss Non-Misleading Headline<br><!-- End of picture text -->
**Table 5** summarizes the descriptive statistics of the three scores for intentions to share

misinformation by study arm. Overall, self-reported likelihood of sharing headlines is low, even for neutral headlines. However, the descriptive statistics are consistent with lower likelihood of sharing misleading headlines than non-misleading headlines. Violin plots depicting the full distribution of scores are also reported in **Appendix H** . .

# **_Effect of Active and Passive Inoculations on Sharing Discernment (H3, H4)_**

**Table 5.** Descriptive statistics of intention to share misinformation by study arm.

||Sha<br>Discer|ring<br>nment|Misle<br>Headline|ading<br>Sharing|Non-Mi<br>Headlin|sleading<br>e Sharing|
|---|---|---|---|---|---|---|
||_M_|_SD_|_M_|_SD_|_M_|_SD_|
|Placebo|0.40|1.37|2.16|1.15|2.56|1.13|
|Control|0.57|1.28|2.09|1.17|2.66|1.09|
|Comprehensive Active Inoculation|0.87|1.43|1.74|1.08|2.61|1.14|
|Brief Active Inoculation|0.64|1.32|1.70|1.07|2.35|1.21|
|Passive Inoculation|0.55|1.31|1.96|1.13|2.51|1.16|

H3 predicted that active inoculations (i.e., game-based) (H3a) and passive inoculations (i.e., infographics) (H3b) will increase discernment in sharing information, compared to the placebo group. **Table** **_6_** summarizes the regression results.

# **Impact of active inoculation on sharing intentions compared to placebo (H3a).**

Contrary to what we found for detecting misinformation in the headlines, only the comprehensive active inoculation arm increased discernment in sharing misinformation compared to the placebo group, partially supporting H3a.<sup>3</sup> Specifically, the comprehensive active inoculation treatment increased sharing discernment by 0.23 _SD_ . Disaggregating the sharing discernment score, we find that the comprehensive inoculation treatment reduces the likelihood

> 3 We did not find differences in outcomes for the headline that uses extreme emotion and the headline that uses a false expert.

23
of sharing misleading headlines by 0.30 _SD_ and does not impact sharing of non-misleading headlines. However, the brief inoculation treatment reduces likelihood of sharing misleading headlines by 0.33 _SD_ and also reduces sharing of non-misleading headlines by 0.26 _SD_ (See **Table 6** , Panel A; **Figure** **_4_** , Panel B). This finding is consistent with the notion that the brief active inoculation may increase overall conservative responding.

# **Impact of passive inoculation on sharing intentions compared to placebo (H3b).**

Contrary to H3b, the passive inoculation group performs similarly to the placebo group, and the difference between the passive inoculation and placebo group is not statistically significant when using Westfall-Young stepdown adjusted _p_ -values to control for the multiple hypothesis testing.

# **Impact of active inoculations on sharing intentions compared to passive inoculation**

**(H4).** H4 predicted that active inoculations (i.e., game-based) will increase discernment in sharing information compared to passive inoculations (i.e., infographics). To test H4, we also run an OLS regression treating the passive inoculation study arm as the base level (see **Table 6** , Panel B). We find that the comprehensive active inoculation improves sharing discernment by 0.24 SD compared to passive inoculation, a pattern driven by a 0.19 _SD_ reduction in sharing misleading headlines. We also find that there is no significant difference between the brief active inoculation and the passive inoculation in sharing discernment. However, this null effect seems to be driven by a 0.14 _SD_ reduction in sharing non-misleading headlines, as the brief inoculation significantly reduced sharing of misleading headlines by 0.22 _SD_ .

24
# **Table 6. Regression results for sharing misinformation** .

||(1)<br>Sharing<br>Discernment|(2)<br>Misleading<br>Headline Sharing|(3)<br>Non-Misleading<br>Headline Sharing|
|---|---|---|---|
|**_Panel A: Placebo as base group_**||||
|Control|-0.126**|0.068|-0.080|
||(0.059)|(0.061)|(0.058)|
||[0.245]|[0.828]|[0.717]|
|Comprehensive Active|0.232***|-0.304***|-0.027|
|Inoculation|(0.060)|(0.058)|(0.057)|
||[0.000]|[0.000]|[0.629]|
|Brief Active Inoculation|0.062|-0.334***|-0.257***|
||(0.056)|(0.057)|(0.058)|
||[0.258]|[0.000]|[0.000]|
|Passive Inoculation|-0.003|-0.118**|-0.120**|
||(0.055)|(0.057)|(0.055)|
||[0.999]|[0.224]|[0.202]|
|Controls|Yes|Yes|Yes|
|Observations|2,851|2,851|2,851|
|R-squared|0.025|0.057|0.028|
|**_Panel B: Passive Inoculation as base group_**||||
|Control|-0.123**|0.186***|0.040|
||(0.059)|(0.059)|(0.059)|
||[0.233]|[0.016]|[0.909]|
|Placebo|0.003|0.118**|0.120**|
||(0.055)|(0.057)|(0.055)|
||[0.999]|[0.224]|[0.202]|
|Comprehensive Active|0.235***|-0.186***|0.093|
|Inoculation|(0.060)|(0.056)|(0.058)|
||[0.001]|[0.005]|[0.115]|
|Brief Active Inoculation|0.065|-0.216***|-0.137**|
||(0.056)|(0.055)|(0.059)|
||[0.439]|[0.000]|[0.089]|
|Controls|Yes|Yes|Yes|
|Observations|2,851|2,851|2,851|
|R-squared|0.025|0.057|0.028|

_Note._ OLS estimates of treatment effects. Control variables are age, education, gender, and vaccination status for COVID-19. Robust standard errors are shown in parenthesis, and Westfall-Young stepdown adjusted p-values in brackets (1,000 bootstrap replications). _*** p<_ 0.01 _, ** p<_ 0.05 _, * p<_ 0.1.

25
<!-- Start of picture text --> N<br>rad<br>o<br>i=<br>=<br>3<br>a<br>oD<br>c<br>=<br>o<br>=<br>a<br>Oo<br>Control Placebo Comprehensive Brief Passive<br>Active Active Inoculation<br>Inoculation Inoculation<br><!-- End of picture text --> <!-- Start of picture text --> o<br>c<br>BS}<br>r=<br>2<br>=n<br>D<br>£<br>G<br>=<br>7)<br><—<br>Control Placebo Comprehensive Brief Passive<br>Active Active Inoculation<br>Inoculation Inoculation<br>[ES Misleading Headline ~=[): Non-Misleading Headline<br><!-- End of picture text -->
# **Impacts on Recommending the Intervention (H5), Confidence, and Difficulty**

H5 poses active inoculations will increase the willingness to recommend the intervention more than passive inoculations. Users in the brief active inoculation arm were more willing to recommend the intervention than the passive inoculation group by 0.15 SD, and users in comprehensive inoculation arm were more willing to recommend the intervention by 0.12 SD (See **Table** **_7_** ). Thus, H5 is supported.

Additionally, both the comprehensive and brief active inoculation treatments lead to more positive attitudes toward the chatbot relative to the placebo group. For instance, participants in the comprehensive and brief active inoculation groups rated that their ability to detect misinformation increased after the chatbot experience by 0.15 SD and 0.20 _SD_ , respectively, compared to the placebo group. They also found the chatbot more challenging. There were no significant differences between the passive inoculation group and the placebo group in any of these self-reported outcomes. See **Figure** **_5_** .

27
**Table 7. Willingness to Recommend Interventions, Attitudes Toward the Chatbot (H5)**

||(1)|(2)|(3)|
|---|---|---|---|
||Confidence on<br>identifying<br>misinformation<br>after the game|Game difficulty|Recommend the<br>game|
|**_Panel A: Placebo as base group_**||||
|Control|-0.055|0.015|-0.069|
||(0.061)|(0.062)|(0.059)|
||[0.828]|[0.828]|[0.828]|
|Comprehensive Active|0.152***|0.286***|0.097*|
|Inoculation|(0.059)|(0.057)|(0.055)|
||[0.029]|[0.000]|[0.140]|
|Brief Active Inoculation|0.201***|0.122**|0.133**|
||(0.058)|(0.060)|(0.052)|
||[0.001]|[0.117]|[0.035]|
|Passive Inoculation|0.007|-0.007|-0.018|
||(0.058)|(0.058)|(0.056)|
||[0.999]|[0.999]|[0.991]|
|Controls|Yes|Yes|Yes|
|Observations|2,851|2,851|2,851|
|R-squared|0.041|0.026|0.092|
|**_Panel B: Passive Inoculation as base group_**||||
|Control|-0.062|0.022|-0.050|
||(0.060)|(0.061)|(0.062)|
||[0.868]|[0.909]|[0.909]|
|Placebo|-0.007|0.007|0.018|
||(0.058)|(0.058)|(0.056)|
||[0.999]|[0.999]|[0.991]|
|Comprehensive Active<br>Inoculation|0.145**|0.294***|0.116**|
||(0.057)|(0.056)|(0.057)|
||[0.044]|[0.000]|[0.098]|
|Brief Active Inoculation|0.194***|0.129**|0.151***|
||(0.056)|(0.059)|(0.055)|
||[0.005]|[0.102]|[0.026]|
|Controls|Yes|Yes|Yes|
|Observations|2,851|2,851|2,851|
|R-squared|0.041|0.026|0.092|

_Note._ OLS estimates of treatment effects. Control variables are age, education, gender, and vaccination status for COVID-19. Robust standard errors are shown in parenthesis, and Westfall-Young stepdown adjusted p-values in brackets (1,000 bootstrap replications). _*** p<_ 0.01 _, ** p<_ 0.05 _, * p<_ 0.1 28
<!-- Start of picture text --> o<br>5<br>Qn<br>”<br>—<br>Control Placebo Comprehensive Brief Passive<br>Active Active Inoculation<br>Inoculation Inoculation<br>Confidence on identifying<br>EE scmmisinformation after bs Game difficulty<br>Ls Recommend the game<br><!-- End of picture text -->
misinformation and not sharing misinformation, than those who missed any of the questions (see **Appendix** **_I_** ). Attentiveness proved to be a relevant factor when using self-paced unsupervised digital tools. Further research is needed to determine the role of attention checks in active inoculations.

Second, to test the stability of the results, we compare the impact of treatments on discernment in judgements and sharing of misinformation with and without covariates (see **Appendix** **_J_** , Table J1). We find that our results are consistent for different model specifications. Third, the evaluation outcomes include the use of edited documents/images, which was directly taught in the active inoculation treatments (the game) but only indirectly addressed in the infographic of the passive inoculation study arm. To ensure that the null impact of the passive inoculation arm is not driven by this difference, we also test the impact of the interventions with only the appeal to emotions and false expert misinformation tactics, which were directly addressed in both the active and passive inoculation arms. **Appendix** **_J_** , Table J2 reports these results, showing that the pattern of findings is consistent with the exclusion of the document manipulation item.

To explore the potential impact of the treatment in Jordan, we included population weights in our analysis to better reflect the population in Jordan. We found that the effect of the active inoculations was no longer significant after these adjustments ( **Appendix** **_J_** , Table J3). This result is not unexpected, as adjusting for population weights can dilute the true impact of the intervention due to the under-representation of offline segments of the population in the sample study. We focused on the online population because this group is more exposed to misinformation, making it a critical and distinct subgroup for our investigation. Moreover, it may be reasonable that effects of a digital chatbot game are greater among populations who generally 30
show interest in such digital tools, with this not being a one-size-fits-all tool Future research can test prebunking inoculation interventions in offline settings.

Finally, to gain deeper insight into the mechanisms driving the observed effects, we examined demographic and relevant sample characteristics for heterogeneity. Given the focus of the chatbot on COVID-19, we also investigated variation in treatment effects across subgroups based on COVID-19 vaccination status. **Appendix** **_K_** presents our findings, which do not find statistically significant interactions, though we have a small sample of unvaccinated respondents in the study.

# **Discussion**

We test the effectiveness of active and passive inoculations on detecting and sharing misinformation. First, we found that both comprehensive and brief active inoculation groups detect misinformation better than the passive inoculation and placebo. Second, we found that both comprehensive and brief active inoculation groups are less willing to share misinformation than the placebo and control groups. Third, we found that the passive inoculation did not significantly increase the ability to detect misinformation or lower the willingness to share misinformation, compared to the placebo. Yet, only the comprehensive active inoculation group is significantly less willing to share misinformation than the infographic group. Moreover, compared to the comprehensive active inoculation group, the brief inoculation group was less accurate in identifying non-misleading headlines. Both active inoculation groups have higher confidence, found the intervention more challenging, and are more willing to recommend the intervention than passive inoculation group. Finally, we demonstrated that our results are robust to potential biases and different specifications, although differential rates in attrition across 31
treatment arms imply that results need to be interpreted as upper bounds of impact with appropriate caution.

These results are consistent with previous evidence that people are better at identifying misinformation after active inoculations, such as games, and that passive inoculations, such as infographics, have smaller or no effect (Basol et al., 2021; Roozenbeek et al., 2020). However, these results were contrary to the meta-analysis findings that both active and passive inoculations improved misinformation detection (Lu et al., 2023). Our results indicate that it is critical to not only educate people about what misinformation looks like but equally what misinformation does _not_ look like to better discern between misleading and credible news. In our study, the brief active inoculation group did not see any examples of non-misinformation and improved only in their classification of misleading headlines. These findings may highlight one mechanism for mitigating overall skepticism, a challenge that some have argued is a consequence of prebunking games (Modirrousta-Galian & Higham, 2023a, p. 3). While other games for inoculation against misinformation, including _Go Viral!_ and _Bad News,_ featured examples of misleading and nonmisleading messages, studies testing the effectiveness of those games did not focus on features that may exacerbate or mitigate potential backfire effects on perceptions of credible information (Basol et al., 2021; Roozenbeek & van der Linden, 2021).

While brief versions of gamified inoculation could be promising to keep participants engaged, as attrition rates were lower in the brief than comprehensive version of the game, showing both misinformation and non-misinformation examples may be crucial to better discerning non-misinformation. To minimize attrition during the intervention, future research may explore other methods of strengthening engagement while maintaining examples to contrast true and ‘fake’ news.

32
Unlike prior research that found passive inoculation effectively improved the ability to detect misinformation (Basol et al., 2021; Lu et al., 2023), we found that passive inoculation was not effective at improving the discernment of misinformation detection. Contrary to the prior research that inoculations did not decrease misinformation sharing intentions (Lu et al., 2023), our results showed that all inoculation treatments significantly decreased intentions to share misinformation, whereas only brief active inoculations decreased the intentions to share nonmisinformation. These results also add to the literature that accuracy judgements and sharing behavior are not always aligned (Pennycook et al., 2021), underscoring the need to treat these outcomes as distinct.

In summary, passive inoculations (i.e., infographics) may fall short for improving the accuracy in detecting misinformation compared to the active inoculations (i.e., games) but they also reached a greater number of participants, as evidenced by lower attrition rates. Further, passive inoculations may still be useful for stopping the spread of misinformation, through reductions in sharing behavior, though this effect did not persist after adjusting for multiple hypothesis testing. Widespread deployment of the infographics may require sufficiently low resources to expose large numbers of individuals at a scale such that these small effects are nonetheless worthwhile in some contexts.

# **Policy Implications**

As policymakers continue to grapple with the spread of mis- and disinformation in an increasingly digitally connected world, the findings presented in this study have important policy implications. This research provides new tools for policymakers and practitioners to effectively boost misinformation detection in the public and stop the spread of misinformation by improving the public’s information literacy and critical thinking skills. In Jordan, the findings are particularly 33
relevant as the latest available data suggests that coverage of COVID-19 vaccinations remained relatively low, under 50% of the population as of March 2023 (Coronavirus Resource Center, 2023). Survey studies highlighted the prevalence of COVID-19 misinformation as a driver of vaccination status in Jordan along with other barriers like perceived financial cost (Sallah et al., 2022) although the government provided the vaccine free of cost and incentivized vaccination through introduction of an online portal that allowed for greater mobility among the vaccinated (Wolters & Abir, 2022). Moreover, the public continues to encounter mis- and dis-information related to vaccination, including growing rumors about potential harms of the recently introduced measles and rubella vaccine for school-aged children in September 2023 (Barakat et al., 2023; Kuttab, 2023).

Beyond vaccination, the inoculation game tested in this study could be effective in stopping the spread of misinformation in other health domains that require active stakeholder engagements to promote accurate health information, such as healthy behaviors to reduce the growing burden of non-communicable diseases (e.g., cardiovascular diseases, diabetes mellitus) and associated risk factors (e.g., hypertension, overweight/obesity, hyperglycemia). For example, misinformation about tobacco, vaping, and e-cigarettes is growing increasingly common online (Al-Rawi et al., 2023; Romer et al., 2020), representing a new threat to global anti-smoking and tobacco control efforts. With such extensions, future research will need to identify the most relevant misinformation techniques and incorporate them into inoculation tools tailored to these domains.

This research also adds to the growing body of evidence of social media interventions (e.g., Donati et al., 2022) demonstrating the value of leveraging digital communication platforms like WhatsApp to implement low-cost interventions and evaluation in environments where resources for program implementation and data collection are limited. In particular, the increasing ease of 34
developing chatbots to reach beneficiaries directly through widely used platforms provides public policy implementers, health service managers, and other stakeholders with an additional channel to give health information to citizens in a user-friendly manner that may better reach those with lower levels of digital literacy than traditional digital resources like websites.

Importantly, the inoculation approach tested in this research is only one of a range of solutions considered effective to combat misinformation (APA, 2023). As such, other tools like the use of accuracy prompts (Pennycook et al., 2021) and carefully designed debunking (Chan et al., 2017) offer complementary approaches to that should be considered as part of a comprehensive set of policy initiatives to combat misinformation.

# **Limitations**

Findings from this study may represent an upper bound of this intervention’s impact due to a number of study limitations and future research is needed to explore the boundary conditions of our findings. Notably, the study only examined the immediate impact of the intervention, as outcomes were measured directly following inoculation. Other research has found evidence of lasting impact from game-based inoculation of up to 3 months with regular training but significant decay without refresher training (Maertens et al., 2021), suggesting a need for additional research to assess medium- and long-term effects our chatbot-based game. Critically, the higher rates of attrition in the comprehensive inoculation arm, compared to the brief and passive inoculation arm, could account for some of the results in our study. Although we largely found balance in demographic characteristics across study arms for those participants who completed the study, it is possible that participants nonetheless differed on unmeasured characteristics that could be relevant to misinformation discernment, such as analytical and critical thinking skills (Arechar et al., 2023). As such, our findings have to be interpreted with this context in mind and future research 35
should seek to mitigate these limitations by testing the intervention with participants that are less likely to drop out and measuring relevant characteristics and including them as covariates in analyses.

It is also important to note that the results of the study should be interpreted in the context of its sampling limitations. As a widely used communication platform in low- and middle-income countries, the use of WhatsApp also offers unique opportunities for scale at a global level. While WhatsApp is a popular global platform, the sample of this study skewed male, young, and educated compared to the national population. Moreover, the ad for the study mentioned the topic of misinformation, which could have impacted self-selection into the study. While the ad also emphasized the game-like experience and highlighted an incentive to participate in an effort to attract a broader audience, results may not be generalizable to a more representative audience. Future research could overcome these limitations by recruiting target populations in offline settings like schools or other public spaces, leveraging broader sampling frames and less biased sampling strategies. More studies are also needed to test the effectiveness of the intervention in other countries with different contexts, particularly other low- and middle-income countries.

# **Conclusion**

Rampant misinformation has become a major global risk exacerbating critical challenges such as climate change and public health (APA, 2023; WEF, 2023). This study contributes to the growing literature that finds that broad-based inoculation by teaching about common manipulation techniques is an effective approach to reducing susceptibility to misinformation. Our study suggests that active inoculation through games may yield stronger effects than more passive approaches, such as infographics, however, the infographic intervention yielded less dropout in the study, highlighting the trade-offs between the two approaches. Moreover, our 36
study sheds light on specific features that might maximize the interventions’ effectiveness. The findings from this study provide encouraging evidence that game-based inoculation should be incorporated into policy initiatives to slow down the spread of misinformation and improve global health. The next step is to test and scale up active inoculation in broader contexts.

37