A Holistic Framework for Analyzing the COVID-19 Vaccine Debate
Abstract
The Covid-19 pandemic has led to infodemic of low quality information leading to poor health decisions.
Combating the outcomes of this infodemic is not only a question of identifying false claims, but also reasoning about the decisions individuals make.
In this work we propose a holistic analysis framework connecting stance and reason analysis, and fine-grained entity level moral sentiment analysis.
We study how to model the dependencies between the different level of analysis and incorporate human insights into the learning process.
Experiments show that our framework provides reliable predictions even in the low-supervision settings.
Introduction
One of the unfortunate side-effects of the Covid19 pandemic is a global infodemic flooding social media with low quality and polarizing information about the pandemic, influencing public perception on it (Tagliabue et al., 2020).
As studies have shown (Montagni et al., 2021), these influences have clear real-world implications, in terms of public acceptance of treatment options, vaccination and prevention measures.
Most computational approaches tackling the
Covid-19 infodemic view it a misinformation detection problem. In other words, they look at identifying false claims and analyzing reactions to them on social media (Hossain et al., 2020; Alam et al., 2021; Weinzierl et al., 2021).
This approach, while definitely a necessary component in fighting the infodemic, does not provide policy makers and health-professionals with much needed information, characterizing the reasons and attitudes that underlie the health and well-being choices individuals make.
Our goal in this paper is to suggest a holistic analysis framework, providing multiple interconnected views of the opinions expressed in
We specifically focus on a timely topic, attitudes explaining vaccination hesitancy.
Figure 1 describes an example of our framework.
Our analysis identifies the stance expressed in the post ( anti-vaccination ) and the reason for it ( distrust of government ).
Given the ideologically polarized climate of social media discussion on this topic, we also aim to characterize the moral attitudes expressed in the text ( oppression ), and how different entities mentioned in it are perceived ( “Biden,
Government” are oppressing, “citizens, us” are oppressed ).
When constructing this framework we tackled three key challenges.
How should these analysis dimensions be operationalized?
While stance prediction is an established NLP task, constructing the space of possible arguments justifying stances on a given topic, and their identification in text, are still open challenges.
We take a human-in-the-loop approach to both problems .
We begin by defining a seed set of relevant arguments based on data-driven studies (Weinzierl et al., 2021; Sowa et al., 2021), where each reason is defined by a single exemplar sentence.
In a sequence of interactions, we use a pre-trained textual inference model to identify paraphrases in a large collection of Covid-19 vaccination tweets, and present a visualization of the results to humans. Humans then perform an error analysis, and either add more sentences to help characterize the existing reason better, or add and characterize additional reasons, based on examples retrieved from the large corpus.
We explain this process in detail in Section 2.
Our morality analysis is motivated by social science studies (Pagliaro et al., 2021; Díaz and Cova, 2021; Chan, 2021) that demonstrate the connection between moral foundation preferences (Haidt and
Graham, 2007; Graham et al., 2009) and Covidrelated health choices. For example, studies show that the endorsement of the fairness and care moral foundations is correlated with trust in science.
To account for fine-grained patterns, we adapt the recently proposed morality-frame formalism (Roy et al., 2021) that identifies moral roles associated with moral foundation expressions in text.
These roles correspond to actor/target roles (similar to agent/patient) and positive or negative polarity, which should be understood in the context of a specific moral foundation.
In Figure 1 “Biden” is the negative actor in the context of Oppression, making him the oppressor.
We explain this formalism in Section 3.
How should the dependencies between these dimensions be captured and utilized?
The combination of stance, reason and moral attitudes provides a powerful source of information, allowing us to capture the moral attitudes expressed in the context of different stances and their reasons.
These connections can also be used to help build expectations about likely attitudes in the context of each stance.
As a motivating example, consider the reason “distrust in government” , which can be associated with the “oppression” moral foundation only when its actor is an entity related to government functions (rather than oppression from Covid-19 illness).
We model these expectation as a probabilistic inference process (Pacheco and Goldwasser, 2021), by incorporating consistency constraints over the judgements made by our model, and predicting the most likely analysis jointly, consisting of all analysis dimensions.
The full model, described using a declarative modeling language, is provided in Section 5.
How can text analysis models be adapted to this highly dynamic domain, without extensive and costly manual annotation?
While our analysis in this paper focuses on a specific issue, vaccination hesitancy, we believe that our analysis framework should be easily adaptable to new issues.
Relying on human insight to characterize and operationalize stance and reason identification is one aspect, that characterizes issue-specific considerations.
Moral Foundation Theory, by its definition, abstracts over specific debate topics, and offers a general account for human morality.
However, from a practical perspective, models for predicting these highly abstract concepts are trained on data specific to a debate topic and might not generalize well.
Instead of retraining the model from scratch, we hypothesize that given an initial model constructed using out-of-domain data, and a small amount of in-domain labeled data, we can obtain acceptable performance by modeling the interaction between reasons, stances and moral foundations.
We study these settings, along with the fully supervised setting in Section 6.
The data, code and tools used in this paper are publicly available * .
Opinion Analysis
To analyze opinions about the Covid-19 vaccine, we model the vaccination stance expressed in each tweet (i.e. pro-vaccine, anti-vaccine, neutral) and the underlying reason behind such stance.
For example, in Figure 1 the tweet expresses an antivaccine stance, and mentions their distrust of the Biden administration as the reason to take this stance.
There are three main challenges involved in this analysis: 1) predicting the stance, 2) constructing the space of possible reasons, and 3) mapping tweets to the relevant reasons.
Stance prediction is an established NLP classification task (Glandt et al., 2021).
However, uncovering latent themes from text automatically remains an open challenge, traditionally approached using noisy unsupervised techniques such as topic models (Zamani et al., 2020b), or by manually identifying and annotating them in text (Hasan and Ng, 2014).
Instead, we combine computational and qualitative techniques to uncover the most frequent reasons cited for pro and anti vaccination stances.
We build on previous health informatics studies that characterized the arguments made against the
Covid-19 vaccine in social media (Wawrzuta et al., 2021).
In this work, researchers come up with a code-book of 12 main themes, frequently used as reasons to refuse or cast doubt on the vaccine.
We [Table] propose an interactive, humans-in-the-loop protocol to learn representations for these 12 initial reasons, ground them in data, evaluate their quality, and refine them to better capture the discussion.
To do this, we build a tool to explore repeating arguments and their reasons in the Covid-19 vaccine debate.
The tool consists of an interactive
Google Colab notebook equipped with a custom
API to query current arguments, ground them in data, and visualize them.
To initialize the system, we use the 12 reasons suggested by Wawrzuta et al. (2021), and represent them using the one-sentence explanation provided.
Our main goal is to ground these reasons in a set of approximately 85,000 unlabeled tweets about the Covid-19 vaccine (details in Section 4).
To map tweets to reasons, we use the similarity between their SBERT embeddings (Reimers and Gurevych, 2019).
The interaction is centered around the operations outlined in Table 1.
Intuitively, the first six operations allow humans to diagnose how reasons map to text, and the last three allows them to act on the result of this diagnosis, by adding and removing reasons, and modifying the phrases characterizing each reason.
We follow a simple protocol during interaction, where three human coders use the operations above to explore the initial reasons.
The coders start by looking at the global picture: the reasons distribution, the 2D visualizations (van der Maaten and
Hinton, 2008) and the silhouette score (Rousseeuw, 1987).
Then, they query the reasons one by one, looking at the word cloud (characterizing the distribution of short phrases over all texts assigned to the reason) and the 10 closest tweets to each reason.
Following these observations, there is a discussion [Table] phase in which the coders follow a thematic analysis approach (Braun and Clarke, 2012) to uncover the overarching themes that are not covered by the current set of reasons, as well as the argumentation patterns that the method fails to identify.
Then, they are allowed to add and remove reasons, as well as explanatory phrases for them in natural language.
Every time a reason or phrase is added or removed, all tweets are reassigned to their closest reasons.
This process was done over two one-hour sessions.
The coders were NLP and Computational
Social Science researchers, two female and one male, between the ages of 25 and 40.
In the first session, the coders focused on adding new reasons and removing reasons that were not prevalent in the data.
For example, they noticed that the initial set of reasons contained mostly antivaccine arguments, and added a positive reason for each negative reason (e.g. government distrust ⇒ government trust ).
In addition to this, they broke down the reason "Conspiracy Theory" into specific conspiracy theories, such as Bill Gates’ micro chip , the vaccine contains fetal tissue , and the vaccine makes you sterile .They
also removed infrequent reasons, such as the swine flu vaccine .
The final set of reasons can be observed in Table 2.
In the second session, the coders focused on identifying the argumentative patterns that were not being captured by the original reason explanations, and came up with overarching patterns to create new examples to improve the representation of the reasons.
For example, in the case of the government distrust reason, the coders found that phrases with strong words were needed (e.g. F the government ), examples that suggested that the government was "good at being bad" (e.g. the government strong record of screwing things up ), and examples with explicit negations (e.g. the government does not work logically ).
Once patterns were identified, each coder contributed a set of 2 to 5 examples, which were introduced to the reason representation.
C ARE /H
ARM : Underlies virtues of kindness, gentleness, and nurturance.
[Table] [Table]
: Generates ideas of justice, rights,
[Table]
autonomy.
[Table]
In Appendix A.1, we include screenshots of the interactive notebook, and tables enumerating the full list derived patterns and phrases.
To visualize the impact of interaction, we also show the overall distribution of reasons before and after interaction, and word clouds for a select set of reasons.
The methodology and tool we developed are broadly applicable for diagnosing NLP models.
Morality Frame Analysis
Moral Foundations Theory (Haidt and Graham, 2007) suggests that there are at least six basic foundations that account for the similarities and recurrent themes in morality across cultures, each with a positive and negative polarity (See Table 3).
To analyze moral perspectives in tweets, we build on the definition of morality frames proposed by Roy et al.
(2021), where moral foundations are regarded as frame predicates, and associated with positive and negative entity roles.
While Roy et al.
(2021) defined different roles types for each moral foundation (e.g. entity causing harm , entity ensuring fairness ), we aggregate them into two general role types: actor and target , each with an associated polarity (positive, negative).
An actor is a “do-er” whose actions or influence results in a positive or negative outcome for the target (the “do-ee”).
For each moral foundation in a given tweet, we identify the “entity doing good/bad” (positive/negative actor) and “entity benefiting/suffering” (positive/negative target).
For example, the statement “We are suffering from the pandemic” expresses harm as the moral foundation, where “pandemic” is a negative actor , and “we” is a negative target (i.e. the entity suffering from the actor’s actions).
There can be zero, one or multiple actors and targets in a given tweet.
Entities can correspond to specific individuals or groups (e.g., I, democrats, people of a given demographic), organizations (e.g., political parties,
CDC, FDA, companies), legislation or other political actions (e.g., demonstrations, petitions), disease or natural disasters (e.g., Covid, global warming), scientific or technological innovations (e.g., the vaccine, social media, the Internet), among others.
We break down the task of predicting morality frames into four classification tasks.
For each tweet, our goal is to predict whether it is making moral judgement or not, and identify its prominent moral foundation.
For each entity mentioned in the tweet, we predict whether it is a target or a role, and whether it has positive or negative polarity.
Data Collection and Annotation
There is no existing corpus of arguments about the Covid-19 vaccine annotated for morality frames and vaccination stance, so we collected and annotated our own.
First, we searched for tweets between April and October of 2021 mentioning specific keywords, such as covid vaccine and vaccine mandate .
The full list of keywords, as well as the procedure to obtain them, can be seen in Appendix A.2.
Then, we created an exclusive web application for annotating our task.
Moral foundation and vaccination stance labels can be annotated directly.
To identify entities, annotators were able to highlight the relevant text spans, and choose its role label (i.e. positive/negative actor or target).
We annotated our dataset using three in-house annotators pursuing a
Ph.D. in Computer Science.
We awarded the annotators $ 0 .
75 per tweet and bonus ( 2 ∗ $0 .
75 = $1 . 5 ) for completing two practice examples.
Our work is
IRB approved, and we follow their protocols.
To ensure quality work, we provided annotators with eight examples covering all six moral foundations and non-moral cases.
Before starting the annotation task, the annotators had to read the instructions, go through the examples, and annotate two practice questions.
The annotation interface, examples and practice questions can be seen in Appendix A.3.
We calculated the agreement among annotators using Krippendorff’s α (Krippendorff, 2004), where α = 1 suggests perfect agreement, and α = 0 suggests chance-level [Figure] agreement.
We found α = 60 .
82 for moral foundations, and α = 78 .
71 for stance.
For roles, we calculated the character by character agreement between annotations.
For example, if one annotator marked “Dr Fauci” as a target in a tweet, and another marked “Fauci”, it was considered to be an agreement on the characters “Fauci” but disagreement on “Dr”.
Doing this, we found α = 83 .
When removing characters marked by all three annotators as "non-role", the agreement dropped to α = 67 . 15 .
We used a majority vote to get moral foundation and vaccination stance labels, and obtained 750 annotated tweets.
Similarly, we defined a text span to be an entity mention E, having a moral role R and polarity P, in a tweet T, if it was annotated as such by at least two annotators.
Our resulting dataset contains(T,E,R,P) tuples.
Statistics can be seen in Table 4. [Figure] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
To evaluate the correlation between the different dimensions of analysis, we calculate the Pearson correlation matrices and present them in Figure 2.
We can interpret reasons as distributions over moral foundations and stances (and vice-versa).
This analysis provides a useful way to explain each of these dimensions.
For example, we see that care/harm is strongly correlated with reasons such as covid is real , the vaccine works , and natural immunity is ineffective .
Other expected trends emerge, such as purity/degradation being highly correlated with vaccine against religion .
To evaluate the modeling advantage of our opinion analysis framework, we look at the correlation between stance, moral foundations and topics extracted in an unsupervised fashion using Latent Dirichlet Allocation (LDA) (Blei et al., 2003). We find that the reasons extracted interactively have higher correlations with both vaccination stance and moral foundations.
The LDA correlation matrices can be seen in Figure 3.
In Table 5 we show the top four reasons for fairness/cheating .
We choose this moral foundation given that it is evenly split among stances and it is active for different reasons.
We show the top two (E,R,P) tuples for each reason.
We can appreciate that while this moral foundation is used by people on both sides, the reasons offered and entities used vary.
On the anti-vax side, authority figures and vaccine trials are portrayed as negative actors, while women and children are portrayed as targets.
On the pro-vax side, Covid and unvaccinated people are portrayed as negative actors, and the general public is portrayed as a target.
In addition to our annotated dataset, we collected a corpus of 85,000 tweets in English mentioning the covid vaccine, uniformly distributed between January and
October of 2021.
These tweets are unlabeled, and are used to ground arguments (Section 2) and to augment data for indirect supervision (Section 5). [Figure] [Figure]
Joint Probabilistic Model
We propose a joint probabilistic model that reasons about the arguments made, their morality frames, stances, reasons, and the dependencies between them.
We implement our model using
DRaiL (Pacheco and Goldwasser, 2021), a declarative modeling framework for specifying deep relational models.
Deep relational models combine the strengths of deep neural networks and statistical relational learning (SRL) to model a joint distribution over relational data.
This hybrid modeling paradigm allow us to leverage expressive textual encoders, and to introduce contextualizing information and model different interdependent decisions.
SRL methods have proven effective to model domains with limited supervision (Johnson and Goldwasser, 2018; Subramanian et al., 2018), and approaches that combine neural networks and SRL have shown consistent performance improvements (Widmoser et al., 2021; Roy et al., 2021).
Following the conventions of statistical relational learning models, we use horn-clauses of the form p
∧ p 1 ∧ ...
∧ p n ⇒ h to describe relational properties.
Each logical rule defines a probabilistic scoring function over the relations expressed in its [Figure]
Base rules /classifiers We define three base rules to score whether a tweet t i has a moral judgment, what is its prominent moral foundation m , and what is its vaccination stance.
[Equation]
To score the moral role of an entity e i mentioned in tweet t i , we write two rules.
The first one scores whether the entity e i is an actor or a target, and the second one scores its polarity (positive or negative).
[Equation]
Note that these rules do not express any dependencies.
They function as base classifiers that map tweets and entities to their most probable labels.
The way an entity is portrayed in a tweet can be highly indicative of its moral foundation.
For example, people are likely to mention children as a negative actor in the context of care/harm .
To capture this, we explicitly model the dependency between an entity, its moral role, and the moral foundation.
[Equation]
As we showed in Section 4, there is a significant correlation between the stance of a tweet with respect to the vaccine debate, and its moral foundation.
For example, people who oppose the vaccine are more likely to express the liberty/oppression moral foundation.
To capture this, we model the dependency between the stance of a tweet and its moral foundation. [Equation]
[Equation]
Dependency between reasons and moral foundations/stances Explicitly modeling the dependency between repeating reasons and other decisions can help us add inductive bias into our model, potentially simplifying the task.
For example, we can enforce the difference between two opposing views that use similar wording, and that could otherwise be treated similarly by a text-based model (e.g. “natural methods of protection against the disease are better than vaccines” vs. ‘vaccines are better than natural methods of protection against the disease” ).
We add two rules to capture this dependency, one between reasons and moral foundations, and one between reasons and stances.
r 7 : Mentions ( t i , r ) ⇒ HasMf ( t i , m ) r 8 : Mentions ( t i , r ) ⇒
VaxStance ( t i , s ) (5)
Hard constraints
To enforce consistency between different decisions, we add two unweighted rules (or hard constraints).
These rules are not associated with a scoring function and must always hold true.
We enforce that, if a tweet is predicted to be moral, then it needs to also be associated to a specific moral foundation.
Likewise, if a tweet is not moral, then no moral foundation should be assigned to it.
[Equation]
Whenever different tweets have the same stance, we include a constraint to enforce consistency between the polarity of different mentions of the same entity.
Roy et al.
(2021) showed that enforcing consistency for mentions of the same entity within a political party was beneficial.
Given the polarization of the Covid-19 vaccine, we use the same rationale.
c 3 : Mentions ( t i , e i ) ∧ Mentions ( t j , e j ) ∧ SameStance ( t i , t j ) ∧ EntPolarity ( e i , p ) ⇒ EntPolarity ( e j , p ) (7)
The weights for each rule w r : p
∧ p 1 ∧ ... ∧ p n ⇒ h measure the importance of each rule in the model and can be learned from data.
For example, when attempting to predict care/harm for a tweet t i , we would like the weight of rule instance IsTweet ( t i ) ⇒
HasMf ( t i , care / harm ) to be greater than the weight of rule instance IsTweet ( t i ) ⇒
HasMf ( t i , loyalty / betrayal ) .
In DRaiL, these weights are learned using neural networks with parameters θ r .
The collection of rules represents the global decision, and the solution is obtained by running a maximum a posteriori (MAP) inference procedure.
Given that horn clauses can be expressed as linear inequalities corresponding to their disjunctive form, the MAP inference problem can be written as a linear program.
DRaiL supports both locally and globally normalized structured prediction objectives.
Throughout this paper, we used the locally normalized objective.
For details about the learning procedure, we refer the reader to the original paper (Pacheco and Goldwasser, 2021).
Learning with low-supervision
To learn DRaiL models in the low-supervision setting, we use an Expectation-Maximization style protocol, outlined in Algorithm 1.
First, we initialize the parameters of base rules using distant supervision classifiers.
For moral foundations, we use the Johnson and Goldwasser (2018) dataset and the Moral Foundation Twitter Corpus (Hoover et al., 2020).
For roles, we use the Roy et al.
(2021) dataset.
For polarity, we combine the Roy et al.
(2021) dataset with the MPQA 3.0 entity sentiment dataset (Deng and Wiebe, 2015).
For vaccination stances, we annotate our 85K unlabeled tweets using a set of prominent antivax and provax hashtags.
Details about these datasets are provided in Appendix A.4.
Once the base rules have been initialized using distant supervision, we turn our attention to learning DRaiL models over the Covid-19 dataset presented in Section 4.
We alternate between MAP inference to obtain training labels (expectation step), and training the neural nets using these labels (maximization step).
We receive an optional parameter k indicating the amount of direct supervision to be used.
When k is provided, k % of the annotated labels are seeded during inference.
Experimental Evaluation
The goal of our framework is to identify morality frames and opinions in tweets by modeling them M ODEL M ORAL /NM [Table]
ORAL F OUND .
A [Table] /T ARGET
E NT .
P OLARITY
V AX S TANCE
[Table]
jointly.
In this section, we perform an exhaustive experimental analysis to evaluate the performance of our model and each of its components.
Experimental settings In DRaiL, each rule r is associated with a neural architecture, which serves as a scoring function to obtain the rule weight w r .
We use BERT-base-uncased (Devlin et al., 2018) for all classifiers.
For the rules that model dependencies (Eqs. 3, 4, 5), we concatenate the CLS token with a one-hot vector of the symbols on the left hand side of the rule (i.e. role, sentiment, stance and reason), before passing it through a classifier.
For rules that have the entity on the left-hand side (Eqs. 2, 3), we use both the tweet and the entity as an input to BERT, using the SEP token.
We trained supervised models using local normalization in DRaiL, and leveraged distant supervision using protocol outlined in Algorithm 1.
In all cases, we used a learning rate of 2 e − 5 , a maximum sequence length of 100 , and AdamW.
In all experiments, we perform 5-fold cross-validation over the annotated dataset and report the micro-averaged results.
Table 6 shows our general results for morality frames and vaccination stance.
We evaluate our base classifiers and show the impact of modeling dependencies using DRaiL. The joint model results in a significant improvement for morality, moral foundation and vaccination stance.
For entities, role and polarity remain stable.
We also measure the impact of explicitly modeling reasons (Eq. 5) and present results in Table 7.
We show the performance for the initial reasons proposed by Wawrzuta et al.
(2021), which are all from the anti-vaccine perspective, and the impact of our two rounds of interaction, expanding and refining reasons (round 1) and augmenting argumentative patterns (round 2).
We find that moral foundations improve from 60.07 to 62.27 and vaccination stance improves from 67.72 to 72.53 after interaction.
[Table]
We show an ablation study in
Table 8.
First, we can see how all dependencies contribute to the performance improvement, roleMF being the most impactful.
We can also see that explicitly modeling morality constraints improves both the morality prediction and the moral foundation prediction, suggesting an advantage to breaking down this decision.
We observe that the stance-polarity constraint does not have a significant impact, but does not hurt performance either, suggesting that our classifiers already capture this information.
Lastly, we can see that the performance for roles and polarity remains stable, potentially because these classifiers have a strong starting point.
Distant supervision
In Figure 4 we evaluate the impact of our indirect supervision protocol by slowly augmenting the amount of direct supervision available.
We can see that by leveraging out of domain-data and dependencies, we can obtain a
M ODEL
M/NM MF A CT /T AR P OLAR .
BERT 69.
84.31 +RoleMF 69.
84.13 +RoleMF+MC 79. 84.13 +StanceMF 69. 84.31 +StanceMF+MC 72. 84.31 +StanceMF+MC+SPC 72.
84.35 +ReasonMF 69. 84.31 +ReasonMF+MC 72. 84.31 +ReasonStance+SPC 69. 83.26 + ALL 81.
[Table]
competitive model using just 25% of the annotated labels, and we can outperform the fully supervised classifiers using 50% of the annotations.
Related Work
Recent studies have noted the prevalence of rumors and misinformation in the context of the Covid19 pandemic (Loomba et al., 2021; Shahi et al., 2021; Lazarus et al., 2021;
Ahmed et al., 2020).
Following this trend, several computational approaches have been proposed to detect misinformation related to Covid in news outlets and social media (Weinzierl and Harabagiu, 2021;
Bang et al., 2021; Serrano et al., 2020; Al-Rakhami and AlAmri, 2020).
In this paper, we take a different approach and look at the problem of identifying opinions surrounding the Covid-19 vaccine, and explicitly modeling the rationale and moral sentiment that motivates them.
Some recent works also look at analyzing arguments about Covid and vaccine hesitancy more broadly.
In most cases, they either take a traditional classification approach for predicting stances (Alliheibi et al., 2021;
Lyu et al., 2021), or use topic modeling techniques to uncover trends in word usage (Skeppstedt et al., 2018; Lyu et al., 2021; Sha et al., 2020; Zamani et al., 2020a).
In contrast, we propose a holistic framework that combines different methodological techniques, including humanin-the-loop mechanisms, classification with distant supervision, and deep relational learning to connect stance prediction, reason analysis and fine-grained entity moral sentiment analysis.
Discussion
We introduce a holistic framework for analyzing social media posts about the Covid-19 vaccine.
We model morality frames and opinions jointly, and show that we can obtain competitive performance.
The main limitation of our work is the size of the annotated dataset studied.
Annotating for morality is a difficult and costly task, as it requires significant domain expertise.
This motivates the need for methods that perform well under limited supervision, and that can leverage external and unlabeled resources.
We took a first step in this direction by combining a wide range of methodological strategies.
Given the amount of data generated daily about Covid, there are broader opportunities for exploiting these resources than what were explored in this paper.
While we provided a preliminary analysis of the correlation between stances, reasons and morality, our current work looks at leveraging this framework to analyze opinions at scale.
We also presented a first step towards interactive exploration of opinions on social media.
While we explored this approach in a limited scenario, there is a lot of potential for using this paradigm for diagnosing NLP models and adapting to new domains.
More research is required to devise protocols and evaluation strategies for this process.  Acknowledgements
We gratefully acknowledge our in-house annotators for their work annotating the dataset presented in this paper. We also thank the anonymous reviewers of this paper for their insightful comments. This project was partially funded by a Microsoft Research Dissertation Grant, a Purdue Graduate
School Summer Research Grant and an NSF CAREER award IIS-
A Appendix A. Reasons and Phrases
Tables 10 and 11 show the full list of phrases for anti-vax and pro-vax reasons.
The interactive task interface is presented in Figures 5 and 6.
Bar plots for reason assignments before and after interaction are shown in Figure 7.
[Figure] [Figure]
A. Data Collection
To create the list of keywords used to collect tweets about the Covid-19 vaccine, we read multiple articles about Covid mentioning vaccination status, vaccine hesitancy, misinformation, vaccine constraints, health issues, religious sentiment and other vaccine-related debates
[Figure]
and made a list of repeating statements.
Then, we consulted three researchers, two in Computational Social Science and one in Psychology
[Figure]
and constructed a list of relevant keywords that are indicative of morally charged discussions.
The full list of keywords can observed in Table
[Figure]
death, covid vaccine military, experimental covid vaccine, covid vaccine authorization, vaccine oppression, vaccine satan, covid vaccine bill gates, covid vaccine side effect, covid vaccine adverse events
A. Data Annotation Task
The steps for annotating tweets using our graphical interface are (See Figure 10).
Select the moral foundation of the text using the checkbox ✓□ .
You can see the definition of each moral foundation by hovering your mouse over them.
If the tweet does not make any moral judgement, check ✓□ "none".
In this case, you don’t have to highlight actor-target polarity.
After selecting any moral foundation other than "none", text highlighting for actor-target role with polarity will be visible below.
If you select a moral foundation other than "none", you can highlight the actor-target polarity.
Choose the color-coded label Positive Actor/Positive Target/Negative Actor/Negative Target to highlight the text with the color of the selected label.
You can see the definition of actor-target-polarity role by hovering your mouse over them.
Highlight words, phrases, or sections of the text for the actor-target role with polarity of the corresponding moral foundation.
If you made any mistakes in highlighting, select the " Unhighlight " button to unhighlight the previously highlighted text.
Finally, click the " Submit " button to submit the task.
We provided eight examples (Figure 11) covering six moral principles and non-moral cases to make our annotation task more understandable.
Annotators could see the explanations for choosing a moral foundation and an. actor-target polarity by clicking the " See Explanation " button. Annotators had to complete two practice examples before starting the real task. If they made any mistake, our practice session provided them the correct result with an explanation.
Figure 12 shows the interface for one of the two practice examples.
A. Out-of-Domain Datasets
For moral foundation prediction, we use the dataset proposed by Johnson and Goldwasser (2018), consisting of 2K tweets by US congress members annotated for the five core moral foundations.
We also use the Moral Foundation Twitter Corpus
Themes
Overarching Patterns
GovDistrust
Add phrases with strong word for distrust “Good at being bad”
Explicit negations
GovTrust
Hedging phrases (sort-of trust)
VaxDanger
Closer connection between vaccine words and danger words (related to sickness, bad effects)
Explit negations
Rhetorical questions
Refusing the vaccine for medical reasons
VaxSafe
Explicit mentions of safety
Explicit negations
CovidFake
Stronger relevant negative words (fake, scam, hoax)
Explicit negations
CovidReal
Trust the science
References to Covid hospitalization on the rise, explicit mentions of hospitals
Explicit negations
VaxOppression
Legal language
Explicit mentions of discrimination and oppression
Sarcasm
VaxNotOppression
Justifying mandates
Freedom to be protected
Criticizing others using “you/people” language, focus freedom on me/my/I
BigPharmaAnti
Stronger words against pharmaceutical companies (corrupt, evil)
Not accountable / irresponsible past behavior
Mentions of negative side-effect of other products (cancer)
BigPharmaPro
Trust science/research and vaccine development process
Language about intent, the vaccine was created to do something good, explicit names of companies
NaturalImmunityPro
The vaccine is not enough
Explicit mentions to population immunity, herd immunity and antibodies
NaturalImmunityAnti
Emphasis on global look, collective entities, society
Natural immunity characterized as dangerous or not effective
Mentions of experts and trusting science
VaxAgainstReligion
I put it in god hands (god is deciding)
Treating pro-vax as another religion
VaxNotAgainstReligion “Religious” in quotes
Bugus exemptions “Where is your faith”
Call to action: get tested/get vaccinated/put a mask on (mentions of compassion)
No religion ask members to refuse vaccine
VaxDoesntWork
Reference to “magic vaccine” “Never developed”, “doesn’t work”
Questions: why are deaths high?
Why is corona not going away?
Why are vaccinated people dying?
VaxWorks “ask a doctor”, consult with an expert
Research on the vaccine is good/has been going on for a long time
Capture differences, e.g. “good trials”
[Table]
[-Full dataset statistics can be osberved in Tab. 
-]
[Table]
[-Care/Harm    
-]
[Table]
(Hoover et al., 2020), consisting of 35k tweets annotated for moral foundations.
The topics across these two datasets span political issues (e.g. gun control, immigration) and events (e.g. Hurricane
Sandy, Baltimore protests).
Given that neither of these two datasets contain examples for the liberty/oppression moral foundation, we curate a small lexicon by looking for synonyms and antonyms of the words liberty and oppression .
Then, we use this lexicon to annotate the congresstweets dataset † .
We annotate a tweet as liberty/oppression if it contains at least four keywords, which results in around 2K tweets.
The derived lexicon for liberty/oppression can be seen in Table
To learn to predict roles, we use the subset of
Johnson and Goldwasser (2018) dataset annotated for roles by Roy et al.
(2021), which contains roughly 3K tweet-entity-role triplets.
For polarity, we combine the Roy et al.
(2021) dataset with the MPQA 3.0 entity sentiment dataset (Deng and
Wiebe, 2015), which contains about 1.6K entitysentiment pairs.
[Table]
[Table]
For stance, we annotate our dataset of 85K unlabeled covid tweets using a set of prominent antivax and provax hashatgs.
For the antivax case, we rely on the hashtags proposed by Muric et al.
For the provax case, we manually annotate hashtags that have a clear provax message, and that are used in at least 50 tweets in our unlabeled dataset.
The full set of hashtags used can be found in Tables 14 and 15.
[Figure]
[Table]
[Table]