metadata: authors: 0: author: Van Der Weide,Roy
count: Middle East and North Africa
docna: 0: docna: Inferring COVID-19 Vaccine Attitudes from
            Twitter Data : An Application to the Arabic Speaking World
docty: Policy Research Working Paper
abstracts: cdata!: This study investigates whether Twitter
            data can be used to infer attitudes towards COVID-19
            vaccination with an application to the Arabic speaking
            world. At first glance, anti-vaccine sentiment estimated
            from Twitter data is surprisingly low in comparison to
            estimates obtained from survey data. Only about 3 percent of
            Twitter accounts in our database are identified as
            anti-COVID-vaccination (compared to 20 to 30 percent of
            survey respondents). This bias is resolved when: (1)
            filtering out accounts belonging to organizations that make
            up a significant share of the discourse on Twitter, and (2)
            adjusting for the fact that the population of Twitter users
            is biased towards more educated individuals. The most
            effective messages on the anti-vaccine side highlight claims
            that the vaccine causes serious life-threatening side
            effects. In the pro-vaccine camp, tweets containing content
            showing public figures receiving the vaccine are found to
            have the largest reach by far.
display_title: Inferring COVID-19 Vaccine Attitudes
            from Twitter Data : An Application to the Arabic Speaking World
listing_relative_url: /research/2022/09/33893768/inferring-covid-19-vaccine-attitudes-twitter-data-application-arabic-speaking-world
topicv3: Health Behavior Change Uptake m1504354 3659,Coronavirus (COVID-19) m1939478 4828
publishtoextweb_dt: 2022-09-06T00:00:00Z
dois: 10.1596/1813-9450-10165
origu: Off of Sr VP Dev Econ/Chief Econ (DECVP)
author_keys: 0: author_key: 677092
model_extractions: Policy Research Working Paper 10165 Inferring COVID-19 Vaccine Attitudes from Twitter Data An Application to the Arabic Speaking World _Roy van der Weide_ Development Economics Development Research Group September 2022
Policy Research Working Paper 10165

# **Abstract**

This study investigates whether Twitter data can be used to infer attitudes towards COVID-19 vaccination with an application to the Arabic speaking world. At first glance, anti-vaccine sentiment estimated from Twitter data is surprisingly low in comparison to estimates obtained from survey data. Only about 3 percent of Twitter accounts in our database are identified as anti-COVID-vaccination (compared to 20 to 30 percent of survey respondents). This bias is resolved when: (1) filtering out accounts belonging to organizations that make up a significant share of the discourse on Twitter, and (2) adjusting for the fact that the population of Twitter users is biased towards more educated individuals. The most effective messages on the anti-vaccine side highlight claims that the vaccine causes serious life-threatening side effects. In the pro-vaccine camp, tweets containing content showing public figures receiving the vaccine are found to have the largest reach by far.

This paper is a product of the Development Research Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The author may be contacted at rvanderweide@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Inferring COVID-19 Vaccine Attitudes from Twitter Data: An Application to the Arabic Speaking World**<sup>**1**</sup>

Roy van der Weide<sup>2</sup> Keywords: COVID-19, health behavior, Twitter, network analysis JEL codes: I12, I15, D85, O33 1 Excellent research assistance was provided by Joey Shea. The authors are grateful to Alexei Abrahams for providing the Twitter data and to Renos Vakis, Mohamad Chatila, and Zeina Afif for sharing the COVID survey data. The authors also thank Johannes Hoogeveen, Ha Minh Nguyen, and Roberta Gatti for their helpful comments and suggestions. Financial assistance was received from the State and Peacebuilding Trust-Fund (SPF). The views expressed here are the authors’ and do not reflect those of the World Bank, its Executive Directors, or the countries they represent.

> 2 The World Bank: rvanderweide@worldbank.org
# **1. Introduction**

Can Twitter data be used to monitor attitudes towards COVID-19 vaccination? How do estimates of vaccination hesitancy rates derived from Twitter data compare to estimates obtained from more conventional survey data? Who are the most effective messengers on Twitter and what are the most effective messages used on either side of the discourse (supporters versus opponents of COVID vaccination)?

We investigate these questions in an application to the Middle East for which we have both conventional survey data on COVID vaccination attitudes and a database of tweets that feature the most prominent COVID hashtag in Arabic spanning the period from mid-December 2020 until mid-April 2021. To identify a given Twitter account’s attitude towards COVID vaccination (for all accounts observed in our database), we adopt the following approach. First, we identify the clusters of user accounts observed in our data. (Clusters are groups of users who are more prone to interacting with each other on Twitter. The resulting clusters, while purely identified by exploiting data on retweet behavior, tend be divided across geographic and ideological lines.) Second, we identify the 50-100 most influential accounts from each cluster. These influencers represent the central nodes in the network of Twitter accounts. Third, we manually inspect the Twitter feeds of these influencers for explicit references towards COVID vaccination that allows us to label them as either pro-vaccination or anti-vaccination (or neutral with respect to COVID vaccination). Fourth, and finally, we iteratively impute the vaccination attitude of other users in our database (i.e. who participated in the discourse on COVID vaccination on Twitter) based on their retweeting of users whom vaccination attitude has been established. We hereby rely on the assumption that a retweet without the addition of any comments is interpreted as an endorsement.

We find that COVID vaccination hesitancy on Twitter in the Middle East is estimated at just below 20 percent, which is of a comparable magnitude as hesitancy rates estimated from survey data, provided that the appropriate filters are applied. At first glance, Twitter-based estimates of hesitancy appear notably downwards biased; when anti-vaccine attitudes are evaluated over the full population of Twitter accounts who engaged in the discourse on COVID vaccination, hesitancy is estimated at around 3 percent. This bias is fully accounted for, however, when we: (1) evaluate hesitancy over Twitter accounts belonging to individuals (i.e. filtering out accounts belonging to organizations who make up a significant share of the discourse on Twitter), and (2) adjust for the fact that the population of Twitter users is biased towards more educated individuals.

With regards to effective messengers and messages in the discourse on COVID vaccination, we find that the most effective tweets on the anti-vaccine side highlight claims that the vaccine causes serious life-threatening side effects -- while on the pro-vaccine side, tweets showing public figures receiving the vaccine are found to be most effective. Finally, state-news accounts are found to receive the overwhelming majority of retweets in our database making them the most effective messengers.

2
The remainder of the paper is organized as follows. Sections 2 and 3 introduce our data and methodology, respectively. The main empirical observations derived from our database of Twitter data on COVID vaccination are presented in Section 4. In Section 5 we contrast our Twitter-based estimates of vaccination hesitancy to estimates obtained from more conventional survey data and rationalize the divergence. Finally, Section 6 concludes.

# **2. Data**

Between December 2020 and April 2021, we used Twitter’s open API to download tweets featuring the most prominent COVID hashtag in Arabic: كورونا#. As our interest is in COVID vaccination attitudes in the region, we filtered the database for vocabulary related to vaccination. Specifically, we compiled a list of 24 vaccination-words in Arabic (shown in Table 1) and identified the sub-sample of tweets that feature at least one of these words.

|**English**|**Arabic**|
|---|---|
|Vaccine|لقاح|
|Vaccine|مصل<br>|
|Treatment|علج|
|Immunization against|تحصين ضد<br>|
|Doctors Syndicate|نقابة<br>أطباء|
|Sinopharm|سينوفارم|
|Vaccination<br>|التلقيح<br>|
|Possible side effects|والعراض الجانبية<br>المحتملة|
|Clinical trials|التجارب<br>السريرية|
|Immunity|المناعة<br>|
|Medicines|أدوية|
|G42 UAE company|شركة<br>G42الماراتية|
|G42 UAE Healthcare Company|شركة<br>G42الماراتية للرعاية<br>الصحية|
|Chinese vaccine|اللقاح الصيني|
|Pfizer vaccine|لقاح فايزر|
|Sputnik vaccine|لقاحسبوتنيك|

Table 1: Vaccination vocabulary Our data is organized into two sub-periods. The first spans from mid-December through to mid-February and contains approximately 730,000 tweets and 230,000 user accounts engaging in the vaccine discourse. Approximately 4000 influencers accounted for 95% of the discourse. The second period runs from late February to early April and contains approximately 500,000 tweets and 190,000 user accounts. About 5100 of these influencers accounted for 95% of the overall discourse.

Twitter shares a select number of tweet and user account attributes, including tweet content, number of likes, number of retweets, user Twitter handle, number of followers. Attributes such as location (i.e. country of residence, nationality), whether the user is an 3
individual or an organization, and the profession of the user (i.e. government official, news, academic, blogger) are not included. We coded these attributes ourselves for the 100 most influential user accounts in our database.

The primary user attribute of interest in our study is the user’s expressed attitude towards COVID vaccination. We identified this attribute for 687 influencers by screening their Twitter feeds and classifying their position with respect to COVID vaccination into one of three categories: (a) anti-vaccination, (b) pro-vaccination, and (c) neutral. User accounts are classified as anti-vaccination if at least one of their tweets expresses overt and explicit rejection of COVID vaccines or vaccination more broadly. Examples of anti-vaccination content includes accounts that point to cases of severe sickness or death post-vaccination, tweets which urge people to rely on alternative immunity boosting techniques to fight COVID instead of taking a vaccine, or content arguing that large pharmaceutical companies were responsible for manufacturing widespread fear of COVID in order to make profits. Conspiratorial content – for example, tweets claiming the vaccine was a plot by Bill Gates to inject the world with microchips – is also classified as anti-vaccination.

Users are classified as pro-vaccination if at least one of their tweets expresses overt and explicit endorsement or support of a COVID vaccine or vaccination program. For example, any content – either written, photographic, or video – indicating that an individual has received a COVID vaccine was classified as pro-vaccination. Other cases include tweets where the account applauds their country’s vaccine rollout, shares content of high-profile figures encouraging vaccination, or shares science-based information about COVID vaccines or vaccination more broadly. Less obvious cases include instances where accounts post and/or retweet news articles about the vaccine. In these instances, additional tweets are manually investigated to ascertain if the account ever explicitly expressed support or rejection of a vaccine or vaccination strategy.

If no overt and explicit sentiment could be identified, then the account is classified as “neutral.” Other user accounts are classified as neutral when they only refer to vaccination as a detail within the context of another issue, where vaccination is only mentioned cursorily. For example, in the case of the trending hashtag “the vaccine is for the Lebanese first” in Lebanon, certain accounts use the hashtag without commenting on vaccination explicitly. Instead, these tweets may comment on racism, corruption, or political divisions in the country, say.

# **3. Methodology**

This section will discuss (1) how we imputed vaccination attitudes for 301,783 user accounts (about 96 percent of the 314,380 users in our database) using the hard-coded vaccination attitudes we observed for the 687 influencers, and (2) how we identified the arguments used by users to support their views towards vaccination.

4
# _Imputing vaccination attitudes_

For the imputation of attitudes towards COVID vaccination, we rely on the assumption that retweets without the addition of qualifications (i.e. excluding quote-retweets) signify endorsements. We hard-code vaccination attitude as a numeric indicator taking the values 1 (pro-vaccination), 0 (neutral), and -1 (hesitant towards vaccination) for a subset of influential user accounts first. Next, we adopt an iterative process where we infer user vaccination attitudes conditional on the attitudes of users that have been established in prior iterations. The attitudes of the initial set of users (i.e., the “seed influencers”) are coded based on a manual inspection of their Twitter feeds. We continue to iterate until convergence is reached, i.e. until we are unable to infer the vaccination attitude for any users for whom we have yet to establish their attitude. Once the attitudes for the “seed influencers” are obtained, the exact iterative steps are:

1. Identify the set of users that satisfy both of the following two criteria: (1) their vaccination attitude has not yet been inferred, and (2) they retweeted at least once a vaccinationrelated tweet in our database from a user account for whom their vaccination attitude has been inferred.

2. For each user identified in step 1, evaluate the mean value over the vaccination attitudes of the user accounts they retweeted.

3. Add the users from steps 1-2 to the users for whom we have estimates of vaccination attitude and return to step 1 (until convergence).

Note that the imputed attitude towards vaccination can attain any value in between -1 and 1 (given that it is obtained as the mean value over the attitudes of endorsed users). If a given user has retweeted 10 users with observed vaccination attitudes consisting of 8 hesitant users, 1 neutral user and 1 pro-vaccination user, then the imputed attitude for this user will take the value of (0.8 * -1) + (0.1 * 0) + (0.1 * 1) = -0.7. The empirical density of imputed attitudes (not reported here) is found to be concentrated at the values -1, 0, and 1, with comparatively little density in between these values. We round the imputed vaccination attitudes to the nearest integer values in order to classify each user into one of the three groups.

Figure 1 plots the interactions between influencers for the first time-period (midDecember 2020 until mid-February 2021) as a network graph. The nodes represent the user accounts, while the “edges” connecting the nodes represent retweets (i.e. two nodes are connected when the two corresponding users retweeted each other). The cluster of nodes colored black shows the set of users who are identified/inferred as being hesitant towards COVID vaccination.

5
<!-- Start of picture text --> as : \ ve.steF ore heves P¢y<br>%e e yd hig °: zs By .<br>~e rat “antr &e Se ¢v f<br>See<br>;; *. oot.yao®tattySiebad* S@\NWekwheB ST! isedguscaeGe +3ee°f<br>&. a, ba N « ° = P2eh ‘ s a ;<br>a?<br>ioe n ae Ta NS fe oe “ou<br>Pon | * 4 % oo Mey 5 Ri . 32 :<br>SD= 4a ya 2 Ba. » oe f<br>ON en gigSonnOPA “aeReeees ksSee.et a<br>5<br>YONI ais “Pees gee hit<br>ee Za ats oat . i yas<br>has eer she” ag at.<br>+ ‘ AE.a eee a6 2 “2 eee<br>e p e Bis ge<br>e<br>we? °<br>ee *% ; * ooo Meategete a a8, * “ : be. -<br>e °‘ an AESoe aoeeo Heed4 aene =<br>. 2 . 99 RR a Seo" ‘<br>YA,s ¢ enHseRIES.$75teedSEbeeons,:*.3<br>Fe “6? “<br>a «6 :<br>Le<br><!-- End of picture text -->
<!-- Start of picture text --> GEE negative MEME neutral MEE positive<br><!-- End of picture text --> <!-- Start of picture text --> GEE negative MEME neutral MEME positive<br><!-- End of picture text -->
<!-- Start of picture text --> Oo<br>S<br>So<br>Oo<br>+<br>°<br>So<br>S<br>28<br>2<br>2<br>ie)<br>ES<br>os<br>o N<br>a2<br>fs<br>Eg<br>So<br>S<br>oO<br>0 10 20 30 40<br>unique rank of (tweet_retweet_count)<br>GS neutral EE negative ME positive<br><!-- End of picture text --> <!-- Start of picture text --> te}<br>N<br>N<br>wo<br>><br>O<br>Cc<br>3)<br>v -<br>wo<br>2<br>oO<br>0 5 10 15<br>log followers<br>——— neutral ——— negative<br>——— positive<br><!-- End of picture text -->
<!-- Start of picture text --> nN<br>oO<br>20<br>o<br>2<br>2<br>ot<br>o<br>N<br>0 5 10 15 20<br>log followers<br>———_ neutral ———— negative ————— positive<br><!-- End of picture text -->
<!-- Start of picture text --> 0 1,000 2,000 3,000 4,000<br>number of retweets<br><!-- End of picture text -->
<!-- Start of picture text --> weet taken down<br>lective & needed<br>n, humor<br>trib<br>0 50,000 100000 150000 200000 250000<br>number of retweets<br><!-- End of picture text -->
<!-- Start of picture text --> NGO / thinktank<br>cademic<br>news<br>ont<br>0 50,000 100000 150000 200000 250000<br>number of retweets<br><!-- End of picture text -->
<!-- Start of picture text --> wo<br>N<br><<br>Sw<br>ae<br>o<br>xo<br>i)<br>o<br>x=<br>a -<br>wo<br>2<br>oO<br>West Bank (FB) Lebanon (FB) Gaza (FB) Lebanon (WFP)<br><!-- End of picture text -->
<!-- Start of picture text --> organization<br>person<br>0 20 40 60<br>number of influencers<br><!-- End of picture text --> <!-- Start of picture text --> GEE negative MEME neutral MEE positive<br><!-- End of picture text --> <!-- Start of picture text --> GEE negative MEME neutral MEE positive<br><!-- End of picture text -->
<!-- Start of picture text --> el<br>~.<br>i<br>&<br>O<br>2o<br>2<br>oO<br><=<br>o<br>N<br>1 2 3 4<br>education<br>——®— Lebanon (FB) ——®— West Bank (FB)<br>——®—- Gaza (FB) ——®— Lebanon (WFP)<br><!-- End of picture text -->
<!-- Start of picture text --> GEE negative MEME neutral MEE positive<br><!-- End of picture text --> <!-- Start of picture text --> GEE negative MEME neutral MEME positive<br><!-- End of picture text -->
# **6. Concluding remarks**

This study investigates whether Twitter data can be used to infer attitudes towards COVID vaccination with an application to the Arabic speaking world. Our objectives are three-fold. First, to estimate the prevalence of anti-vaccine attitudes using Twitter data. Second, to identify which messengers and what arguments are found to be most effective. Third, to assess how Twitterbased estimates of vaccine hesitancy compare to estimates obtained from survey data.

The following findings stand out. At first glance, anti-vaccine sentiment estimated from Twitter data is surprisingly low in comparison to estimates obtained from survey data. Only about 3 percent of Twitter accounts in our database are identified as anti-COVID-vaccination (compared to 20 to 30 percent of survey respondents). The most effective messages on the anti-vaccine side highlight claims that the vaccine causes serious life-threatening side effects. In the pro-vaccine camp, tweets containing content showing public figures receiving the vaccine are found to have the largest reach by far. This observation sits well with the earlier study by Alatas et al. (2019), which finds that Twitter endorsements of vaccination by celebrities are most effective. The study furthermore finds that celebrity endorsements can influence beliefs about vaccination. With respect to messengers in our application to the MENA region, we find that state-news accounts receive the overwhelming majority of retweets in our database.

To explain why our estimates of anti-vaccine attitudes are visibly lower than survey-based estimates, which are found to range approximately between 20 to 30 percent, we consider three candidate sources of bias. First, Twitter accounts include both individuals and organizations, whereas survey respondents consist of individuals only. The organizations in our sample are primarily news organizations, NGOs, private companies, law firms, government entities, and international bodies – all of which are more likely to express overtly positive attitudes towards vaccination. Individuals, on the other hand, are more likely to hold anti-vaccine attitudes and express them explicitly online. Second, Twitter users are a selective sub-sample of the population biased towards higher levels of education, higher income and increased political engagement. Third, the vaccine rollout during the period in which the data for our sample was collected was slow and uneven. Therefore, users may not have developed strong views on vaccination during this period, nor did they have the incentive to express these views on Twitter.

Most of the difference between the Twitter-based and survey-based estimates of COVID vaccine hesitancy can be accounted for by restricting Twitter accounts to those belonging to individuals. Using the survey data, we furthermore find that vaccine hesitancy declines with the respondent’s level of education. Restricting the survey samples to individuals who completed a secondary degree or higher we obtain survey-based estimates of vaccine hesitancy that are of the same magnitude as Twitter-based estimates of hesitancy (among accounts belonging to individuals).

Two possible directions for future work include: (a) extending this analysis to other regions of the world, starting with the Twitter discourse on COVID vaccination in English (which will be a magnitude larger in volume), (b) extending the analysis to other social media platforms, 17
including the discourses observed on public Facebook pages, and (c) studying the effectiveness of COVID vaccination campaigns.

# **Bibliography**

Alatas, V., Chandrasekhar, A., Mobius, M., Olken, B. and C. Paladines (2019), “When celebrities speak: A nationwide Twitter experiment promoting vaccination in Indonesia”, _NBER Working Paper 25589_ Eurofound (2021), “Living, working and COVID-19 (Update April 2021): Mental health and trust decline across EU as pandemic enters another year”, _April 2021 Factsheet_ Fridman A, Gershon R, and A. Gneezy (2021), “COVID-19 and vaccine hesitancy: A longitudinal study.” _PLOS ONE_ 16(4): e0250123. https://doi.org/10.1371/journal.pone.0250123 Germani, F. and N. Biller-Andorno (2021), “The anti-vaccination infodemic on social media: A behavioral analysis.” _PLOS ONE_ 16(3):e0247642. https://doi.org/10.1371/journal.pone.0247642 Jamison A, Broniatowski D., Smith M., et al. (2020), “Adapting and Extending a Typology to Identify Vaccine Misinformation on Twitter.” _American Journal of Public Health_ . 110(S3): S331S339. doi:10.2105/AJPH.2020.305940 Kata, A. (2010), “A postmodern Pandora's box: Anti-vaccination misinformation on the Internet.” _Vaccine._ 28(7); 1709-1716 Meadows C, Tang L. and W. Liu (2019), “Twitter message types, health beliefs, and vaccine attitudes during the 2015 measles outbreak in California.” _Am J Infect Control_ . 47(11):1314-1318. doi: 10.1016/j.ajic.2019.05.007. Epub 2019 Jun 29. PMID: 31266661.

Puri, N., Coomes, E., Haghbayan, H. and K. Gunaratne (2020), “Social media and vaccine hesitancy: new updates for the era of COVID-19 and globalized infectious diseases.” _Human Vaccines & Immunotherapeutics,_ 16:11, 2586-2593, DOI: <u>10.1080/21645515.2020.1780846</u> Schmidt A., Zollo F., Scala A., Betsch C. and W. Quattrociocchi (2018), “Polarization of the vaccination debate on Facebook.” _Vaccine_ . 36(25):3606-3612 Troiano G. and A. Nardi (2021), “Vaccine hesitancy in the era of COVID-19 **.** ” _Public Health_ . Volume 194, 245-251.

18