Automatic Identification of 5 C Vaccine Behaviour on Social Media (2022)

Abstract: Monitoring vaccine behaviour through social media can guide health policy. We present a new dataset of 9471 tweets posted in Australia from 2020 to 2022, annotated with sentiment toward vaccines and also 5C, the five types of behaviour toward vaccines, a scheme commonly used in health psychology literature. We benchmark our dataset using BERT and Gradient Boosting Machine and show that jointly training both sentiment and 5C tasks (F1=48) outperforms individual training (F1=39) in this highly imbalanced data. Our sentiment analysis indicates close correlation between the sentiments and prominent events during the pandemic. We hope that our dataset and benchmark models will inform further work in online monitoring of vaccine behaviour. The dataset and benchmark methods are accessible online.

## introduction the development of effective and safe vaccines has been shown as one of the most successful means to mitigate the spread of covid-19 disease around the globe. as with any other vaccination program, covid-19 intervention in any country depends on public acceptance and vaccine uptake. this has to be done by a large proportion of society to attain herd immunity, which is estimated to range from 67% to 95% @xcite @xcite . however, vaccine hesitancy has been identified as one of the top 10 challenges to global health by the world health organization in 2019 @xcite ). vaccine hesitancy refers to the delay in acceptance or refusal of vaccination by the public despite the availability of vaccination services @xcite . this behaviour has been significantly prominent towards covid-19 vaccines as they differ from previous vaccines in many respects: accelerated development, novel techniques used, potential side effects, uncertainty regarding the size and extent of their effectiveness, and limited production compared to the demand @xcite . australia began its mass vaccination program in late february 2021 but was initially hampered by low vaccine uptake due to mistrust of vaccine effectiveness and the government, the blood clotting syndrome (tts) associated with the astrazeneca product, shipment delays, and misinformation @xcite . despite the fact that the present uptake is high (> 90% two doses for over 16 years), the coverage varies across age, jurisdictions, and the number of doses (australian government: department of health and aged care, 2022 august [cited 14 @xcite ). high coverage is primarily driven by travel desires (and travel vaccination requirements) and the need to mitigate risk. as new variants of sars-cov-2 are emerging and existing immunity waning in a short period of time, it is likely that people will need to get multiple booster doses. consequently, identifying immediate and future public behaviour toward vaccination is important for public health authorities to combat possible challenges to reaching and maintaining herd immunity. the 5c model provides five measures (confidence, complacency, constraints, calculation and collective responsibility) for assessing an individual's psychological reasoning towards vaccination @xcite . confidence associates the trust in vaccine effectiveness, safety, and the system that delivers it. people with low confidence mistrust the healthcare system, fall for misinformation, believe in conspiracies, and doubt the benefit of vaccines. complacency exists when vaccination is viewed as a low priority or when vaccine-preventable diseases are not of a concern. high complacency correlates with low uptake of a vaccine. constraints refer to the structural and psychological barriers to getting vaccinated. for example, geographical constraints in accessibility, limited language, and health literacy, and cost may postulate high constraints. calculation defines the engagement in extensive information searching, which may lead to lower vaccination willingness arising with exposure to a high volume of antivaccination content. collective responsibility asserts an individual's willingness to protect others by getting vaccinated and contributes to herd immunity. see most works in identifying vaccine behaviour use surveys that are costly, time-consuming, and don't scale to a large population. the use of social media for expressing vaccine-related opinion provides a great opportunity to use online social monitoring tools to guide health policy-making for vaccine adoption. there are few recent studies that adapt the 5c scheme for online health monitoring. @xcite focus on the analysis of tweets rather than building predictive models, fues @xcite 's uses 1794 tweets from scandinavian users and manually categorise them into the 5c categories, and @xcite focuses on the vaccine trials and uses unsupervised methods. our contributions are as follows: 1) we provide the first large-scale vaccine behaviour dataset in english, annotated with both 5c and sentiment; 2) we present two benchmark models and show the challenges of 5c predictive models given the highly imbalanced data; and 3) we analyse the data showing that changes in 5c distribution is not uniform across regions, indicating opportunities for targeted health messaging. we make the dataset and benchmarks available hoping to impact future work in online vaccine behaviour monitoring based on the 5c framework. ## related work the 5c model has been widely applied to examine covid-19 vaccination behaviour. in 2020, @xcite estimated hong kong nurses' intention to receive covid-19 vaccine using the 5c model and examines the correlation of their vaccine behaviour to previous influenza vaccination. @xcite applied the 5c scale to investigate psychological reasoning behind the previous uptake of measles and flu vaccines by adults in the united states and their intention to get covid-19 vaccination for themselves and their children. @xcite studied the psychological drivers of vaccination intention in university students across the netherlands, belgium and portugal, using the 5c model. @xcite implemented the 5c model to investigate older adults' vaccination behaviour in the united kingdom over the first year of the pandemic. @xcite examined the levels and predictors of acceptance of an approved covid-19 vaccine in eight western countries by utilizing the 5c model. @xcite applied the 5c model to identify vaccine hesitancy among chronic disease patients availing care in a primary health facility in india. previous studies examined covid-19 vaccination behavior utilizing traditional surveys @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . however, such surveys are often costly in their design and implementation, time-consuming, produces limited data and represent comparatively short-term situation. recently, twitter has been increasingly applied in research concerning public attitude towards vaccination due to its advantages of availability of a large amount of real-time posts without any costs, ease of access and public searching facility. in spite of these advantages, there remains a gap in australia for twitter-based covid-19 vaccination research. we found just one study @xcite , conducted during 2020, which addresses australian public opinion towards covid-19 vaccine solely from australian twitter users. sentiment analysis of this study has shown that the majority of people expressed positive emotions towards vaccine with trust and anticipation as the most prominent behaviors associated with it, while fear being the top negative emotion. an investigation of twitter posts from 10 countries, including australia, has revealed that more information about vaccines' safety and the expected side effects may increase public positive attitudes towards vaccination @xcite . similarly, sentiment analysis from 4 million tweets across several nations, including australia, has found the prevalence of vaccine hesitancy and objections outweighs vaccine interests @xcite . various other studies have assessed covid-19 vaccination behavior using twitter data from a specific country or a particular region. these include, studies based on posts from the united states @xcite @xcite , the united kingdom @xcite , japan @xcite , china @xcite , canada @xcite , africa @xcite . other studies address covid-19 vaccination behaviour at a global scale @xcite @xcite . along with sentiment analysis, some studies have implemented the 5c model to examine covid-19 vaccination emotions as well as attitudes towards other vaccines in tweets. @xcite constructed a multiple linear regression model to examine positive attitudes towards vaccines across ten countries. in their model, among other methods, the positive 5c categories were applied to identify the covariates. @xcite applied a topic modeling approach to investigate the mistrust in covid-19 vaccination based on english and french tweets. they then used the 5c scale to categorize the topics and found that all mistrusts fell into the confidence category. fues @xcite applied the 5c model to map relevant predictors for several vaccination behaviours in scandinavian twitter users. similar to our work, they manually labelled each tweet according to the 5c scale. however, unlike our dataset, their dataset was multi-labeled, as each tweet was assumed to have multiple 5c categories. they did not, however, include covid-19 vaccination as a specific vaccination category in their analysis. ## results our results assert that the largest f1 value corresponds to positive sentiments (88%) from the sentiment task (task 1) and the constraint class (79%) in predicting the 5c task (task 2) ( for the combined sentiment and 5c prediction task (task 3), our prediction depicts that the negative constraints (80%) and positive calculation and collective responsibility categories (59% for both) have the highest f1 measure ( we stress that gbm uses boosting, so it oversamples difficult instances, for example, less frequent ones, and thus can handle class imbalance in an indirect way to some extent. however, both gbm and bert were to be affected by the class imbalance. when the weights of the low-frequency categories were increased to fix this issue, they reduced the accuracy of other categories, and hence we did not implement class weights in the models. ## conclusion we presented the first large dataset for online monitoring vaccination behaviour in australia, which is annotated by public sentiments towards vaccination and their psychological reasoning by the 5c scale. our analysis has shown a close correlation between the sentiments of the tweets and the prominent events during the pandemic. our analysis of vaccination behaviour from this dataset showed model p% r% f1% bert 53 38 48 gbm 36 28 32 our work provides a proof of concept in the application of 5c scales to monitor vaccination behaviour using social media and can be extended to other domains such as facebook and google trends.