COVID -19 Vaccine Misinformation in Middle Income Countries (2023)

Abstract: This paper introduces a multilingual dataset of COVID-19 vaccine misinformation, consisting of annotated tweets from three middle-income countries: Brazil, Indonesia, and Nigeria. The expertly curated dataset includes annotations for 5,952 tweets, assessing their relevance to COVID-19 vaccines, presence of misinformation, and the themes of the misinformation. To address challenges posed by domain specificity, the low-resource setting, and data imbalance, we adopt two approaches for developing COVID-19 vaccine misinformation detection models: domain-specific pre-training and text augmentation using a large language model. Our best misinformation detection models demonstrate improvements ranging from 2.7 to 15.9 percentage points in macro F1-score compared to the baseline models. Additionally, we apply our misinformation detection models in a large-scale study of 19 million unlabeled tweets from the three countries between 2020 and 2022, showcasing the practical application of our dataset and models for detecting and analyzing vaccine misinformation in multiple countries and languages. Our analysis indicates that percentage changes in the number of new COVID-19 cases are positively associated with COVID-19 vaccine misinformation rates in a staggered manner for Brazil and Indonesia, and there are significant positive associations between the misinformation rates across the three countries.

## introduction affluent countries have enjoyed ample supply of covid-19 vaccines but distribution to low-and middle-income countries were slow. the void created by a slow vaccine roll out may create a space for misinformation to further flourish. misinformation on social media is already documented to contribute to vaccine hesitancy in developed countries @xcite , but comparatively few studies have focused beyond high income settings @xcite . given that middle income countries (mic) make up 75% of the world's population and 62% of the world's poor @xcite , studying vaccine misinformation in mic is important to preparing for future pandemics especially because global uptake to vaccines is needed to end pandemics @xcite . importantly, the impact of misinformation on vaccine hesitancy extends beyond covid-19 in that there has been a precipitous decline in childhood vaccination rates in mic @xcite . the residual effects of vaccine misinformation on vaccine hesitancy presents a global health problem for containing infectious diseases. in this study, we focus on three middle-income countries: brazil, indonesia, and nigeria, which have the majority of their population using social media @xcite and where vaccine confidence has eroded in recent years @xcite . in addition, each of these three countries are population hubs from three distinct regions in the world. while the majority of the population in these countries use social media @xcite , social media usage is still projected to grow in these mic countries whereas usage has plateaued in high-income countries @xcite . this presents a unique opportunity to study the impact of social media on vaccine misinformation in mic. the development of an automated system that can detect vaccine misinformation will allow for epidemic surveillance in three heavily populated countries in distinct parts of the world. this will allow government agencies, health agencies, and researchers to monitor misinformation and coordinate intervention strategies to deter emerging infectious diseases. in addition, the automatic system for misinformation detection allows for identification of shared misinformation themes across three distinct countries/regions, which would enable health organizations to tailor health messages. in this work, we make two main contributions. firstly, we have curated a new multilingual dataset of covid-19 vaccine misinformation from mic, where social media misinformation is understudied. this dataset foot_0 is the first of its kind in several aspects: it is geolocated, in multiple languages, carefully curated, covers a wide time range from 2020 to 2022, and includes not only domain-expert annotations of misinformation, but also the identified themes based on a previous study of globally circulating covid-19 vaccine information on online platforms @xcite . secondly, utilizing this dataset, we have trained models that effectively detect misinformation and its associated themes, outperforming competitive baselines, including gpt-3 models. we apply our best models in a large scale study to analyze the trend of misinformation in the three mic between 2020 and 2022. our analysis yields interesting findings regarding the relationship between covid-19 cases, vaccine misinformation rates, and variations across the three countries. ## related works while there have been studies on vaccine misinformation, no prior study has examined misinformation across countries or studied shared misinformation themes across mic. studies on vaccine misinformation utilizing twitter data tend to be country-specific, predominantly focused on the united states @xcite , restricted to the english language only @xcite , or lack specified geolocations @xcite . moreover, these studies are conducted during limited periods in 2021 @xcite , or in 2020 @xcite . similarly, most previous works introducing new datasets related to covid-19 misinformation primarily focus on english content from specific target locations @xcite @xcite @xcite @xcite , while @xcite released an arabic tweet dataset covering countries in the arab region. studies on vaccine misinformation are important as misinformation exposure contributes to increased vaccine hesitancy and reduced behavioral intention to get vaccinated @xcite . media plays a prominent role in spreading misinformation. for instance, a longitudinal study on mother's attitudes towards mmr vaccines reveals a link between media-published misinformation, their perception of vaccine safety, and vaccination rates among children in the united kingdom @xcite . similarly, negative press in local media in the uk was associated with a decline in mmr vaccinations @xcite . the impact of misinformation on vaccination coverage is not limited to europe; it is observed globally. a well-documented case is polio in central and west africa, where in the late 1990s and early 2000s misinformation about the polio vaccine spread within muslim communities in northern nigeria, benin, burkina faso, cameroon, central african republic, and other neighboring countries @xcite , leading to leaders refusing vaccination and subsequent outbreaks of the disease @xcite . ## dataset we collect all publicly available twitter posts in brazil, indonesia, and nigeria that contain vaccinerelated terms from january 2020 to december 2022 (@xmath0,809,231). we use the hydration process @xcite where we use brandwatch api to scrape ids of tweets that contained predefined vaccine-related terms, then apply twitter api to retrieve tweet contents. brandwatch enables us to scrape tweets specific to geolocations associated with the three focal countries. there are no language exclusions. in addition, we include "shot" as a search term because it is often used to describe vaccine, but we set limits to reduce data noise, particularly from references to parties and alcohol that are synonymous with the term "shot". our complete list of search query is listed in appendix a.1. in addition, we include country-specific terms including "vacina," "injecao," and "zé gotinha" for brazil; and "vaksin" and "suntik" for indonesia per the oxford languages word for vax @xcite . for data annotation, we employ quantitative content analysis (qca) in communication research @xcite where a representative sample of the data is drawn and on which two or more trained coders (i.e., annotators) apply a codebook protocol that contains all the variables for annotation and their definitions. similar to @xcite , prior to coding the q yes (%) no (%) uncertain (%) total q1 3,666 (61.6) 2,286 (38.4) -5,952 q2 655 (17.9) 3,011 (82.1) -3,666 q3 1,119 (37.2) 1,596 (53.0) 296 (9.8) 3,011 q4a 186 (16.6) 933 (83.4) -1,119 q4b 539 (48.2) 580 (51.8) -1,119 q4c 26 (2.3) 1,093 (97.7) -1,119 q4d 412 (36.8) 707 (63.2) -1,119 q4e 106 (9.5) 1,013 (90.5) -1,119 q4f 93 (8.3) 1,026 (91.7) -1,119 q4g 272 (24.3) 847 (75.7) -1,119 q4h 29 (2.6) 1,090 (97.4) -1,119 to apply qca, we randomly sample 5,500 tweets, stratified by the distribution of tweets collected in each of the three focal countries. two communication graduate students are trained as coders to examine the tweets for: (q1) relevance to vaccine (yes/no), (q2) mentions of specific non-covid vaccine (such as the mmr vaccine) (yes/no), (q3) presence of misinformation (yes/no/uncertain). content deemed to be misinformation is labeled further for its themes based on a pre-identified list of covid-19 vaccine misinformation themes @xcite , which included misinformation pertinent to ( q4a q4g ) vaccine morbidity or mortality (0.88); and (q4h) vaccine alternatives (0.89). upon determination that human coding is reliable, the trained coders independently code the remaining posts. finally, tweets annotated with "uncertain" for misinformation are reviewed by our expert public health panel who make the final determination. to balance the data for misinformation, we identify more tweets that contain misinformation, resulting in the final annotated dataset of @xmath1,952. ## covid-19 vaccine misinformation models in this section, we describe how we develop our covid-19 vaccine misinformation models and present the experiment results. ## analysis for the final models, the best-performing model for each question, discovered during the crossvalidation process ( ## conclusion in this work, we present a multilingual dataset of tweets from three middle income countries annotated with their relevance to covid-19 vaccines, the presence of misinformation, and themes of misinformation. our method leveraging domainspecific pre-training and text augmentation demonstrate performance improvement for detecting misinformation and its themes. we also show applications of our covid-19 vaccine misinformation system by conducting extensive analysis on a large corpus of tweets from 2020 to 2022. distributed lag model indicates that % changes in the number of new covid-19 cases increase misinformation rate over time for brazil and indonesia. we find positive association in misinformation rates between brazil, indonesia, and nigeria. additionally, our qualitative content analysis on tweets shows that vaccine misinformation can be contingent on local factors. we believe our dataset would allow public health communicators and researcher to discern the types of misinformation that are prevalent in each country and facilitate targeted communication strategies that are specifically tailored to local circumstances.