A Holistic Framework for Analyzing the COVID -19 Vaccine Debate (2022)

Abstract: The Covid-19 pandemic has led to infodemic of low quality information leading to poor health decisions. Combating the outcomes of this infodemic is not only a question of identifying false claims, but also reasoning about the decisions individuals make. In this work we propose a holistic analysis framework connecting stance and reason analysis, and fine-grained entity level moral sentiment analysis. We study how to model the dependencies between the different level of analysis and incorporate human insights into the learning process. Experiments show that our framework provides reliable predictions even in the low-supervision settings.

## introduction one of the unfortunate side-effects of the covid-19 pandemic is a global infodemic flooding social media with low quality and polarizing information about the pandemic, influencing public perception on it @xcite . as studies have shown @xcite , these influences have clear real-world implications, in terms of public acceptance of treatment options, vaccination and prevention measures. most computational approaches tackling the covid-19 infodemic view it a misinformation detection problem. in other words, they look at identifying false claims and analyzing reactions to them on social media @xcite @xcite . this approach, while definitely a necessary component in fighting the infodemic, does not provide policy makers and health-professionals with much needed information, characterizing the reasons and attitudes that underlie the health and well-being choices individuals make. our goal in this paper is to suggest a holistic analysis framework, providing multiple interconnected views of the opinions expressed in * equal contribution text. we specifically focus on a timely topic, attitudes explaining vaccination hesitancy. figure 1 describes an example of our framework. our analysis identifies the stance expressed in the post (anti-vaccination) and the reason for it (distrust of government). given the ideologically polarized climate of social media discussion on this topic, we also aim to characterize the moral attitudes expressed in the text (oppression), and how different entities mentioned in it are perceived ("biden, government" are oppressing, "citizens, us" are oppressed). when constructing this framework we tackled three key challenges. our morality analysis is motivated by social science studies @xcite @xcite that demonstrate the connection between moral foundation preferences @xcite and covidrelated health choices. for example, studies show that the endorsement of the fairness and care moral foundations is correlated with trust in science. to account for fine-grained patterns, we adapt the recently proposed morality-frame formalism @xcite that identifies moral roles associated with moral foundation expressions in text. these roles correspond to actor/target roles (similar to agent/patient) and positive or negative polarity, which should be understood in the context of a specific moral foundation. in figure 1 "biden" is the negative actor in the context of oppression, making him the oppressor. we explain this formalism in section 3. ## opinion analysis to analyze opinions about the covid-19 vaccine, we model the vaccination stance expressed in each tweet (i.e. pro-vaccine, anti-vaccine, neutral) and the underlying reason behind such stance. for example, in figure 1 the tweet expresses an antivaccine stance, and mentions their distrust of the biden administration as the reason to take this stance. there are three main challenges involved in this analysis: 1) predicting the stance, 2) constructing the space of possible reasons, and 3) mapping tweets to the relevant reasons. stance prediction is an established nlp classification task @xcite . however, uncovering latent themes from text automatically remains an open challenge, traditionally approached using noisy unsupervised techniques such as topic models @xcite , or by manually identifying and annotating them in text @xcite . instead, we combine computational and qualitative techniques to uncover the most frequent reasons cited for pro and anti vaccination stances. we build on previous health informatics studies that characterized the arguments made against the covid-19 vaccine in social media @xcite . in this work, researchers come up with a code-book of 12 main themes, frequently used as reasons to refuse or cast doubt on the vaccine. we show_reasons() lists the current list of reasons (e.g. government distrust, natural immunity.) show_closest_tweets(reason, k) lists the k tweets closest to a given reason, based on their embedding similarity. wordcloud(reason) renders a word cloud to visualize the arguments associated to a given reason, based on bigram and trigram tf-idf features. show_assignments(threshold) renders a bar plot showing the assignment of tweets to reasons, based on embedding similarity. an optional threshold can be used to limit assignments. tsne(threshold) renders a visualization of the reason clusters in a 2d map. threshold is optional. silhouette_score(threshold) measures the overlapping degree between clusters. threshold is optional. add_reason(reason, phrase) adds a new reason with a phrase that characterizes it in natural language remove_reason(reason) removes a given reason add_phrase(reason, phrase) adds an additional phrase to an existing reason. ## morality frame analysis moral foundations theory @xcite suggests that there are at least six basic foundations that account for the similarities and recurrent themes in morality across cultures, each with a positive and negative polarity (see to analyze moral perspectives in tweets, we build on the definition of morality frames proposed by @xcite , where moral foundations are regarded as frame predicates, and associated with positive and negative entity roles. while @xcite defined different roles types for each moral foundation (e.g. entity causing harm, entity ensuring fairness), we aggregate them into two general role types: actor and target, each with an associated polarity (positive, negative). an actor is a "do-er" whose actions or influence results in a positive or negative outcome for the target (the "do-ee"). for each moral foundation in a given tweet, we identify the "entity doing good/bad" (positive/negative actor) and "entity benefiting/suffering" (positive/negative target). for example, the statement "we are suffering from the pandemic" expresses harm as the moral foundation, where "pandemic" is a negative actor, and "we" is a negative target (i.e. the entity suffer-ing from the actor's actions). there can be zero, one or multiple actors and targets in a given tweet. entities can correspond to specific individuals or groups (e.g., i, democrats, people of a given demographic), organizations (e.g., political parties, cdc, fda, companies), legislation or other political actions (e.g., demonstrations, petitions), disease or natural disasters (e.g., covid, global warming), scientific or technological innovations (e.g., the vaccine, social media, the internet), among others. we break down the task of predicting morality frames into four classification tasks. for each tweet, our goal is to predict whether it is making moral judgement or not, and identify its prominent moral foundation. for each entity mentioned in the tweet, we predict whether it is a target or a role, and whether it has positive or negative polarity. ## data collection and annotation there is no existing corpus of arguments about the covid-19 vaccine annotated for morality frames and vaccination stance, so we collected and annotated our own. first, we searched for tweets between april and october of 2021 mentioning specific keywords, such as covid vaccine and vaccine mandate. the full list of keywords, as well as the procedure to obtain them, can be seen in appendix a.2. then, we created an exclusive web application for annotating our task. moral foundation and vaccination stance labels can be annotated directly. to identify entities, annotators were able to highlight the relevant text spans, and choose its role label (i.e. positive/negative actor or target). we annotated our dataset using three in-house annotators pursuing a ph.d. in computer science. we awarded the annotators @xmath00.75 = $1.5) for completing two practice examples. our work is irb approved, and we follow their protocols. to ensure quality work, we provided annotators with eight examples covering all six moral foundations and non-moral cases. before starting the annotation task, the annotators had to read the instructions, go through the examples, and annotate two practice questions. the annotation interface, examples and practice questions can be seen in appendix a.3. inter-annotator agreement we calculated the agreement among annotators using krippendorff's α @xcite , where α = 1 suggests perfect agreement, and α = 0 suggests chance-level agreement. we found α = 60.82 for moral foundations, and α = 78.71 for stance. for roles, we calculated the character by character agreement between annotations. for example, if one annotator marked "dr fauci" as a target in a tweet, and another marked "fauci", it was considered to be an agreement on the characters "fauci" but disagreement on "dr". doing this, we found α = 83.46. when removing characters marked by all three annotators as "non-role", the agreement dropped to α = 67.15. ## joint probabilistic model we propose a joint probabilistic model that reasons about the arguments made, their morality frames, stances, reasons, and the dependencies between them. we implement our model using drail (pacheco and goldwasser, 2021), a declarative modeling framework for specifying deep relational models. deep relational models combine the strengths of deep neural networks and statistical relational learning (srl) to model a joint distribution over relational data. this hybrid modeling paradigm allow us to leverage expressive textual encoders, and to introduce contextualizing information and model different interdependent decisions. srl methods have proven effective to model domains with limited supervision @xcite , and approaches that combine neural networks and srl have shown consistent performance improvements @xcite . following the conventions of statistical relational learning models, we use horn-clauses of the form p 0 ∧ p 1 ∧ ... ∧ p n ⇒ h to describe relational properties. each logical rule defines a probabilistic scoring function over the relations expressed in its body and head. base rules/classifiers we define three base rules to score whether a tweet t i has a moral judgment, what is its prominent moral foundation m, and what is its vaccination stance. to score the moral role of an entity e i mentioned in tweet t i , we write two rules. the first one scores whether the entity e i is an actor or a target, and the second one scores its polarity (positive or negative). note that these rules do not express any dependencies. they function as base classifiers that map tweets and entities to their most probable labels. ## experimental evaluation the goal of our framework is to identify morality frames and opinions in tweets by modeling them train all rules locally using y gold 8: end while jointly. in this section, we perform an exhaustive experimental analysis to evaluate the performance of our model and each of its components. ## related work recent studies have noted the prevalence of rumors and misinformation in the context of the covid-19 pandemic @xcite @xcite . following this trend, several computational approaches have been proposed to detect misinformation related to covid in news outlets and social media @xcite @xcite . in this paper, we take a different approach and look at the problem of identifying opinions surrounding the covid-19 vaccine, and explicitly modeling the rationale and moral sentiment that motivates them. some recent works also look at analyzing arguments about covid and vaccine hesitancy more broadly. in most cases, they either take a traditional classification approach for predicting stances @xcite , or use topic modeling techniques to uncover trends in word usage @xcite @xcite . in contrast, we propose a holistic framework that combines different methodological techniques, including humanin-the-loop mechanisms, classification with distant supervision, and deep relational learning to connect stance prediction, reason analysis and fine-grained entity moral sentiment analysis. ## discussion we introduce a holistic framework for analyzing social media posts about the covid-19 vaccine. we model morality frames and opinions jointly, and show that we can obtain competitive performance. the main limitation of our work is the size of the annotated dataset studied. annotating for morality is a difficult and costly task, as it requires significant domain expertise. this motivates the need for methods that perform well under limited supervision, and that can leverage external and unlabeled resources. we took a first step in this direction by combining a wide range of methodological strategies. given the amount of data generated daily about covid, there are broader opportunities for exploiting these resources than what were explored in this paper. while we provided a preliminary analysis of the correlation between stances, reasons and morality, our current work looks at leveraging this framework to analyze opinions at scale. we also presented a first step towards interactive exploration of opinions on social media. while we explored this approach in a limited scenario, there is a lot of potential for using this paradigm for diagnosing nlp models and adapting to new domains. more research is required to devise protocols and evaluation strategies for this process.