Disentangled Learning of Stance and Aspect Topics for Vaccine Attitude Detection in Social Media (2022)

Abstract: Building models to detect vaccine attitudes on social media is challenging because of the composite, often intricate aspects involved, and the limited availability of annotated data. Existing approaches have relied heavily on supervised training that requires abundant annotations and pre-defined aspect categories. Instead, with the aim of leveraging the large amount of unannotated data now available on vaccination, we propose a novel semi-supervised approach for vaccine attitude detection, called VADet. A variational autoencoding architecture based on language models is employed to learn from unlabelled data the topical information of the domain. Then, the model is fine-tuned with a few manually annotated examples of user attitudes. We validate the effectiveness of VADet on our annotated data and also on an existing vaccination corpus annotated with opinions on vaccines. Our results show that VADet is able to learn disentangled stance and aspect topics, and outperforms existing aspect-based sentiment analysis models on both stance detection and tweet clustering.

## introduction the aim of vaccine attitude detection in social media is to extract people's opinions towards vaccines by analysing their online posts. this is closely related to aspect-based sentiment analysis in which both aspects and related sentiments need to be identified. previous research has been largely focused on product reviews and relied on aspectlevel sentiment annotations to train models @xcite , where aspect-opinions are extracted as triples @xcite , polarized targets @xcite or sentiment spans @xcite . however, for the task of vaccine attitude detection on twitter, such a volume of annotated data is barely available @xcite the astrazeneca one is rough for up to 48 hours; after that you may still be a bit swollen but you'll basically feel fine. i've had that and the virus, and the vaccine is far less unpleasant. have felt for the past 24 hours that i've been run over by three double decker buses after the astrazeneca vaccine yesterday morning. starting to feel a little normal now but it's not been nice! there are some very interesting ties between this vaccines creators and the eugenics movement which is concerning considering it's mainly been promoted as a vaccine for poor folks in the third world. 2021). this scarcity of data is compounded by the diversity of attitudes, making it difficult for models to identify all aspects discussed in posts @xcite . as representative examples, consider the two tweets about personal experiences for vaccination at the top of figure 1 . the two tweets, despite addressing a common aspect (vaccine side-effects), express opposite stances towards vaccines. however, the aspect and the stances are so fused together that the whole of the tweets need to be considered to derive the proper labels, making it difficult to disentangle them using existing methodologies. additionally, in the case of vaccines attitude analysis, there is a wide variety of possible aspects discussed in posts, as shown in the bottom of figure 1 , where one tweet ironically addressed vaccine side-effects and the second one expressed instead specific political concerns. this is different from traditional aspect-based sentiment analysis on product reviews where only a small number of aspects need to be pre-defined. the recently developed framework for integrating variational auto-encoder (vae) @xcite and independent component analysis (ica) @xcite sheds light on this problem. vae is an unsupervised method that can be used to glean information that must be retained from the vaccine-related corpus. meanwhile, a handful of annotations would induce the separation of independent factors following the ica requirement for prior knowledge and inductive biases @xcite . to this end, we could disentangle the latent factors that are either specific to the aspect or to the stance, and improve the quality of the latent semantics learned from unannotated data. we frame the problem of vaccine attitude detection as a joint aspect span detection and stance classification task, assuming that a tweet, which is limited to 280 characters, would usually only discuss one aspect. in particular, we extend a pretrained language model (lm) by adding a topic layer, which aims to model the topical theme discussed in a tweet. in the absence of annotated data, the topic layer is trained to reconstruct the input message built on vae. given the annotated data, where each tweet is annotated with an aspect span and a stance label, the learned topic can be disentangled into a stance topic and an aspect topic. the stance topic is used to predict the stance label of the given tweet, while the aspect topic is used to predict the start and the ending positions of the aspect span. by doing so, we can effectively leverage both unannotated and annotated data for model training. to evaluate the effectiveness of our proposed model for vaccine attitude detection on twitter, we have collected over 1.9 million tweets relating to covid vaccines between february and april 2021. we have further annotated 2,800 tweets with both aspect spans and stance labels. in addition, we have also used an existing vaccination corpus foot_0 in which 294 documents related to the online vaccination debate have been annotated with opinions towards vaccination. our experimental results on both datasets show that the proposed model outperforms existing opinion triple extraction model and bert qa model on both aspect span extraction and stance classification. moreover, the learned latent aspect topics allow the clustering of user attitudes towards vaccines, facilitating easier discovery of positive and negative attitudes in social media. the contribution of this work can be summarised as follows: our work is related to three lines of research: aspect-based sentiment analysis, disentangled representation learning, and vaccine attitude detection. aspect-based sentiment analysis (absa) aims to identify the aspect terms and their polarities from text. much work has been focusing on this task. the techniques used include conditional random fields (crfs) @xcite , bidirectional long short-term memory networks (bilstms) @xcite , convolutional neural networks (cnns) @xcite , attention networks @xcite , denselstms @xcite , nestedl-stms (moniz and krueger, 2017), graph neural networks @xcite and their combinations @xcite @xcite , to name a few. @xcite framed this task as text span detection, where they used text spans to denote aspects. the same annotation scheme was employed in @xcite , where intra-word attentions were designed to enrich the representations of aspects and predict their polarities. @xcite formalized the task as a sequence labeling problem under a unified tagging scheme. their follow-up work @xcite explored bert for end-toend absa. @xcite modified this task by introducing opinion terms to shape the polarity. a similar modification was made in @xcite to extract aspect-opinion pairs. positionaware tagging was introduced to entrench the offset between the aspect span and opinion term @xcite . more recently, instead of using pipeline approaches or sequence tagging, @xcite adapted syntactic dependency parsing to perform aspect and opinion expression extraction, and polarity classification, thus formalizing the task as structured sentiment analysis. ## methodology the goal of our work is to detect the stance expressed in a tweet (i.e., 'pro-vaccination', 'antivaccination', or 'neutral'), identify a text span that indicates the concerning aspect of vaccination, and cluster tweets into groups that share similar aspects. to this end, we propose a novel latent representation learning model that jointly learns a stance classifier and disentangles the latent variables capturing stance and aspect respectively. our proposed vaccine attitude detection (vadet) model is firstly trained on a large amount of unannotated twitter data to learn latent topics via masked language model (lm) learning. it is then fine-tuned on a small amount of twitter data annotated with stance labels and aspect text spans for simultaneously stance classification and aspect span start/end position detection. the rationale is that the inductive bias imposed by the annotations would encourage the disentanglement of latent stance topics and aspect topics. in what follows, we will present our proposed vadet model, first under the masked lm learning and later extended to the supervised setting for learning disentangled stance and aspect topics. vadet in the masked lm learning we insert a topic layer into a pre-trained language model such as albert, as shown in figure 2 , allowing the network to leverage pre-trained information while fine-tuned on an in-domain corpus. we assume that there is a continuous latent variable z involved in the language model to reconstruct the original text from the masked tokens. we retain the weights of a language model and learn the latent representa- outputs topic layer [cls] very grateful to those at oxford. i've got my first #covid19 vaccine. tion during the fine-tuning. more concretely, the topic layer partitions a language model into lower layers and higher layers denoted as ψ and θ, respectively. the lower layers constitute the encoder that parameterizes the variational posterior distribution denoted as q ϕ (z|ψ(w)), while the higher layers reconstruct the input tokens, which is referred to as the decoder. the objective of vae is to minimize the kldivergence between the variational posterior distribution and the approximated posterior. this is equivalent to maximizing the evidence lower bound (elbo) expressed as: here, @xmath0, w 1:n ], since the special classification embedding w cls is automatically prepended to the input sequence @xcite , w h denotes the reconstructed input. following (kingma and welling, 2014), we choose a standard gaussian distribution as the prior, denoted as p(z), and the diagonal gaussian distribution z ∼ n (µ ϕ (ψ(w)), σ 2 ϕ (ψ(w))) as the variational distribution. the decoder computes the probability of the original token given the latent variable sampled from the encoder. we use the memory scheme @xcite to concatenate z and ψ(w), making the latent representation com-patible for higher layers of the language model. then the latent presentation z is passed to θ to reconstruct the original text. vadet with disentanglement of aspect and stance one of the training objectives of vaccine attitude detection is to detect the text span that indicates the aspect and to predict the associated stance label. existing approaches rely on structured annotations to indicate the boundary and dependency between aspect span and opinion words @xcite , or use a two-stage pipeline to detect the aspect span and the associated opinion separately @xcite . the problem is that the opinion expressed in a tweet and the aspect span often overlap. to mitigate this issue, we instead separate the stance and aspect from their representations in the latent semantic space, that is, disentangling latent topics learned by vadet into latent stance topics and latent aspect topics. a recent study in disentangled representation learning @xcite shows that unsupervised learning of disentangled representations is theoretically impossible from i.i.d. observations without inductive biases, such as grouping information @xcite or access to labels @xcite ). as such, we extend our model to a supervised setting in which disentanglement of the latent vectors can be trained on annotated data. figure 3 outlines the overall structure of vadet in the supervised setting. on the right hand side, we show vadet learned from the annotated aspect text span [w a : w b ] under masked lm learning. the latent variable z a encodes the hidden semantics of the aspect expression. we posit that the aspect span is generated from a latent representation with a standard gaussian distribution being its prior. the elbo for reconstructing the aspect text span is: on the left hand side of figure 3 , we train vadet on the whole sentence. the input to vadet is formalized as: '[cls] text'. instead of mapping an input to a single latent variable z, as in masked lm learning of vadet, the input is now mapped to a latent variable decomposing into two components, [z s , z w ], one for the stance and another for the aspect. we use a conditionally factorized gaussian prior over the latent variable z w ∼ p θ (z w |w a:b ), which enables the separation of z s and z w since the diagonal gaussian is factorized and the conditioning variable w a:b is observed. we establish an association between z w and z a by specifying p θ (z w |w a:b ) to be the encoder network of q ϕ (z a |w a:b ), since we want the latent semantics of aspect span to encourage the disentanglement of attitude in the latent space. in other words, the prior of z w is configured as the approximate posterior of z a to enforce the association between the disentangled aspect in sentence and the de facto aspect. as a result, the elbo for the original text is written as the annotation of the stance label provides an additional input. to exploit this inductive bias, we enforce the constraint that z s participates in the generation of @xcite , which follows an approximate posterior q ϕ (z s |ψ(w [cls] )). we place the standard gaussian as the prior over z s ∼ n (0, i) and obtain the elbo since the variational family in eq. 1 are gaussian distributions with diagonal covariance, the joint space of [z s , z w ] factorizes as @xcite . assuming z w to be solely dependent on ψ(w 1:n ), we obtain the elbo for the entire input sequence: note that the expectation term can be decomposed into the expectation term in eq. 3 and eq. 4 according to the decoder structure. for the full derivation, please refer to appendix a. finally, we perform stance classification and classification for the starting and ending position over the aspect span of a tweet. we use negative log-likelihood loss for both the stance label and aspect span: ## experiments we present below the experimental setup and evaluation results. (e.g., pfizer, astrazeneca, and moderna). our final dataset comprises 1.9 million english tweets collected between february 7th and april 3rd, 2021. we randomly sample a subset of tweets for annotation. upon an initial inspection, we found that over 97% of tweets mentioned only one aspect. as such, we annotate each tweet with a stance label and a text span characterizing the aspect. in total, 2,800 tweets have been annotated in which 2,000 are used for training and the remaining 800 are used for testing. the statistics of the dataset is listed in baselines we compare the experimental results with the following baselines: bertqa @xcite : a pre-trained language model well-suited for span detection. with bertqa, attitude detection is performed by first classifying stance labels then predicting the answer queried by the stance label. the text span is configured as the ground-truth answer. we rely on its hugging-face foot_2 @xcite implementation. we employ albert @xcite as the backbone language model for both bertqa and vadet. aste @xcite : a pipeline approach consisting of aspect extraction @xcite and sentiment labelling @xcite . evaluation metrics for stance classification, we use accuracy and macro-averaged f1 score. for aspect span detection, we follow @xcite in adopting exact match (em) accuracy of the starting-ending position and macro-averaged f1 score of the overlap between the prediction and ground truth aspect span. for tweet clustering, we follow @xcite and @xcite and use the normalized mutual information (nmi) metric to measure how the clustered group aligns with ground-truth categories. in addition, we also report the clustering accuracy. ## conclusions in this work, we presented a semi-supervised model to detect user attitudes and distinguish aspects of interest about vaccines on social media. we employed a variational auto-encoder to encode the main topical information into the language model by unsupervised training on a massive, unannotated dataset. the model is then further trained under a semi-supervised setting that leverages annotated stance labels and aspect spans to induce the disentanglement between stances and aspects in a latent semantic space. we empirically showed the benefits of such an approach for attitude detection and aspect clustering over two vaccine corpora. ablation studies show that disentangled learning and unsupervised pre-training are important to effective vaccine attitude detection. further investigations on the quality of the disentangled representations verify the effectiveness of the disentangled factors. while our current work mainly focuses on short text of social media data where a sentence is assumed to discuss a single aspect, it would be interesting to extend our model to deal with longer text such as online debates in which multiple arguments or aspects may appear in a single sentence.