Classifying COVID -19 Vaccine Narratives (2023)

Abstract: Vaccine hesitancy is widespread, despite the government’s information campaigns and the efforts of the World Health Organisation (WHO). Categorising the topics within vaccine-related narratives is crucial to understand the concerns expressed in discussions and identify the specific issues that contribute to vaccine hesitancy. This paper addresses the need for monitoring and analysing vaccine narratives online by introducing a novel vaccine narrative classification task, which categorises COVID-19 vaccine claims into one of seven categories. Following a data augmentation approach, we first construct a novel dataset for this new classification task, focusing on the minority classes. We also make use of fact-checker annotated data. The paper also presents a neural vaccine narrative classifier that achieves an accuracy of 84% under cross-validation. The classifier is publicly available for researchers and journalists.

## introduction vaccination is one of the most effective public health interventions, but it is essential that immunisation programs are able to achieve and sustain high vaccine uptake rates. overcoming vaccine hesitancy, which refers to the delay in the uptake or refusal of vaccines, is a major challenge @xcite and the who has named it one of the top ten threats to global health in 2019 @xcite . vaccine hesitancy is a complex and context specific phenomenon, varying across time, place and even vaccines @xcite . it could be caused by various factors such as concerns about side effects, costs, and misinformation. although social media platforms like twitter, facebook, and youtube have taken actions to limit the spread of misinformation, simply identifying and removing misinformation from platforms is not enough, as the concerns of the vaccine-hesitant citizens also need to be monitored and responded to. consequently, fact-checkers and other professionals need analytical tools that help them to better monitor misinformation, vaccine hesitancy, vaccine-related debates and their narratives. topic analysis of narratives about vaccines could be used for this purpose, however, a large manual effort is required, due to the lack of a vaccinerelated topic classifier. for example, @xcite gather over 14 million vaccine-related posts from twitter, instagram, and facebook to research vaccine-related narratives. the posts are categorised into six topics based on a novel typology designed to capture the ways narratives are framed. however, manual analysis was feasible on only a small sample of 1,200 posts, which, given the small scales, leaves significant gaps in the understanding and tackling of vaccine hesitancy. guided by these needs, the novel contributions of this paper are in: ## related work since the outbreak of the covid-19 pandemic and accompanying infodemic, large-scale monolingual and multilingual datasets have been collected from different social media platforms in order to intervene and combat the spreading of covid-19related disinformation @xcite @xcite @xcite , with vaccines being a commonly included topic in these datasets. as the importance of understanding and tackling covid-19 vaccination hesitancy grew, increasing efforts have been made to analyse vaccine narratives and discourses, the dissemination of false claims and the anti-vaccine groups on social media. this has resulted in the construction of a number of covid-19 vaccine-focused datasets, without @xcite or with annotations about veracity (e.g., true or false information) @xcite , sentiment (e.g., positive, negative or neutral) @xcite , stance (e.g., pro-or anti-vaccine) @xcite @xcite or topic category (e.g., vaccine development or side effects) @xcite . the datasets, consequently, can be used to facilitate the research on covid-19 vaccine-related online information from different aspects, including factchecking, sentiment analysis, stance detection, and topic analysis. topics or themes discussed in the vaccine-related narratives and online debates are an essential dimension. state-of-the-art methods for automatic topic analysis typically fall under one of these categories: topic modelling @xcite @xcite , clustering @xcite @xcite , and inductive analysis @xcite . topic modelling, represented by latent dirichlet allocation (lda) @xcite , is the most commonly used approach at present @xcite @xcite . clustering methods for topic discovery have been applied to text representations @xcite or networks @xcite . for instance, k-means @xcite has been used to cluster the average word embeddings of vaccine narratives @xcite or to test a humanderived topic typology @xcite . after constructing a co-occurrence topic network with hashtags as nodes, the louvain method @xcite is used to extract clustering from the graph @xcite . the above methods are unsupervised, resulting in no control on the model generation. therefore, extra work is normally involved in discovering and labelling the topics. in contrast, inductive analysis relies on experts to analyse the raw textual data and derive topics or themes @xcite @xcite . for instance, @xcite categorise anti-vaccine tweets into twelve conversation themes, such as negative health impacts, pharmaceutical industry and religion. @xcite identify twenty-two narrative tropes (e.g., corrupt elites and vaccine injury) and sixteen rhetorical strategies (e.g., brave truthteller and appropriating feminism) in anti-vaccine and covid-denialist social media posts. besides the above work specific to anti-vaccine contents, general covid-19 vaccine narratives on social media were categorised by fact-checkers and researchers at first draft @xcite ) as belonging to one of six topics, as shown in a potential drawback of inductive analyses is that the amount of data that can be analysed by the human experts is significantly smaller than the volumes analysed through the automatic topic modelling and clustering methods. to overcome this problem, @xcite create a list of unique keywords for each theme during inductive analysis, which are then used to automatically categorise more posts based on keyword matching. in this paper, we explore machine learning and deep learning methods for automatic vaccine narrative classification according to the topics proposed by @xcite . to the best of our knowledge, this is the first paper to frame online vaccine narrative categorisation as a classification task. in that respect, topic description examples conspiracy (cons) known or novel conspiracies and conspiracy theories involving vaccines or their development bill gates: we need to depopulate the planet. also bill gates: save your life with my vaccine. development, provision and access (dpa) the ongoing progress or challenges concerning the development, testing and provision of vaccines as well as the access to vaccines oxford coronavirus vaccine triggers immune response. liberty/freedom (lf) civil liberties and personal freedom considerations surrounding vaccines and vaccination policies states have authority to fine or jail people who refuse coronavirus vaccine, attorney says. morality, religiosity and ethics (mre) moral, ethical and religious concerns around vaccines kanye west praises trump, hammers planned parenthood, likens covid vaccine to 'mark of the beast'. politics and economics (pe) political, economic or business developments related to vaccines scientists worry about political influence over coronavirus vaccine project. safety, efficacy and necessity (sen) safety and efficacy of vaccines, including the perceived necessity of vaccines who warns coronavirus vaccine alone won't end pandemic: 'we cannot go back to the way things were'. in addition, we introduce a new, seventh category that encompasses claims related to animal vaccines (animalvac). the motivation is to recognise or filter out animal vaccine-related posts, which are also captured by keyword-based data collection methods that are typically used for collecting vaccine-related social media posts (e.g., using keywords such as vaccine or vaccines). thus, this paper regards the vaccine narrative categorisation task as a seven-way classification problem, with six topics pertaining to covid-19 human vaccination and one additional topic for animal vaccination. ## predictive model we evaluate feature-based and transformer-based models that are pre-trained with out-of-domain and in-domain data, and models that combine classification and topic modelling. ## results ## analysis 7.1 evaluation of data augmentation we analyse (1) whether our newly collected posts improve the performance on the minority classes in fd data; (2) whether the introduction of the animalvac class impacts the performance on the six human-related vaccine classes. data split for the first purpose, we construct two training sets (training set(imbalanced) and training set(balanced)) and a test set (test set(six-class)). the data of the six topics except for mre in the fd data is randomly split in the ratio of 7:3 in the case of training set(imbalanced) and test set(six-class). since the mre class only consists of seven posts in the fd dataset, we include them in the test set(six-class) only. the newly collected mre posts are randomly split in the same ratio as above to complete the training set(imbalanced) and test set(six-class). the training set(balanced) is the combination of training set(imbalanced) and the rest of the new posts we collected during data augmentation. to contrast the performance before and after the introduction of the new category animalvac, we randomly split the data points in the animalvac class into two parts (7:3) and add them into training set(balanced) and test set(six-class) respectively, that is, training set(seven-class) and test set(seven-class). ## conclusion this paper proposed a novel seven-way classification task for categorising online vaccine narratives. we augmented an existing six-class dataset semiautomatically, leading to a more balanced data distribution and the inclusion of an additional seventh category of posts related to animal vaccines. we experimented with strong baseline models and our best model cantm-covid achieves an accuracy score of 0.84 using 5-fold cross-validation. we also show that data augmentation of minority classes helps to produce better models, without significantly impacting the performance on the remaining classes. moreover, the addition of the new animal vaccine category does not significantly influence model performance on the original six human vaccine related classes. in our discussion, we highlighted the main challenges of this task and the current limitations of our model. future work will focus on addressing some of those challenges, including development of models capable of dealing with longer posts. last but not least, our vaccine narratives classifier is made available through an api for reproducibility reasons. we believe this is a significant contribution towards understanding and tracking online debates around vaccine safety and hesitancy.