Disentangling Aspect and Stance via a S iamese Autoencoder for Aspect Clustering of Vaccination Opinions (2023)

Abstract: Mining public opinions about vaccines from social media has been increasingly relevant to analyse trends in public debates and to provide quick insights to policy-makers. However, the application of existing models has been hindered by the wide variety of users’ attitudes and the new aspects continuously arising in the public debate. Existing approaches, frequently framed via well-known tasks, such as aspect classification or text span detection, make direct usage of the supervision information constraining the models to predefined aspect classes, while still not distinguishing those aspects from users’ stances. As a result, this has significantly hindered the dynamic integration of new aspects. We thus propose a model, namely Disentangled Opinion Clustering (DOC), for vaccination opinion mining from social media. DOC is able to disentangle users’ stances from opinions via a disentangling attention mechanism and a Swapping-Autoencoder, and is designed to process unseen aspect categories via a clustering approach, leveraging clustering-friendly representations induced by out-of-the-box Sentence-BERT encodings and disentangling mechanisms. We conduct a thorough experimental assessment demonstrating the benefit of the disentangling mechanisms and cluster-based approach on both the quality of aspect clusters and the generalization across new aspect categories, outperforming existing methodologies on aspect-based opinion mining.

## introduction mining public opinions about vaccines from social media has been hindered by the wide variety of users' attitudes and the continuously new aspects arising in the public debate of vaccination @xcite . the most recent approaches have adopted holistic frameworks built on morality analysis @xcite or neural-based models predicting users' stances on different aspects of the online debate @xcite . so far, these frameworks have been frequently framed via well-known tasks, such as aspect classification or text span detection, that use supervision to train text classifiers. however, such a direct usage of the supervision information has constrained the models to predefined aspect classes and restricted their flexibility in generalising to opinions with aspects never seen before (e.g., new moral issues or immunity level). to mitigate this limitation, some of the most promising approaches have been devised as supervised models generating clustering-friendly representations @xcite . these have recently shown promising results on open-domain tasks when combined with pre-trained language models (plm) thanks to their flexibility, generalisation, and need for minimal tweaks @xcite . however, despite the improved capabilities in capturing the overall text semantics, existing models for text clustering @xcite @xcite , still struggles to distinguish between the mixed users' stances and aspects on vaccination, and as a result, they often generate clusters that do not reflect the novel aspects of interest. as an illustrating example, consider the tweets "mrna vaccines are poison" and "the pfizer vaccine is safe", that the majority of existing methodologies are prone to cluster into different groups due to the opposite stances manifested, despite the fact that both of them are targeting safety issues. to address the aforementioned problem, we posit that a model should be able to (i) disentangle the stance from the aspect discussed, and simultaneously (ii) use the generated representations in a framework (e.g., clustering) that ease the integration of aspects never seen before. we thus propose a novel representation learning approach, called the disentangled opinion clustering (doc) model, which performs disentan-gled learning @xcite ) via text autoencoders @xcite , and generates clustering-friendly representations suitable for the integration of novel aspects foot_0 . the proposed model, doc, learns clusteringfriendly representations through a denoising autoencoder @xcite driven by outof-the-box sentence-bert embeddings @xcite , and disentangles stance from opinions by using the supervision signal to drive a disentangled cross-attention mechanism and a swapping autoencoder @xcite . we conducted an experimental assessment on two publicly available datasets on vaccination opinion mining, the covid-moral-foundation (cmf) @xcite and the vaccination attitude detection (vad) corpora @xcite . we first assessed the quality of the disentangled representation in generating aspect-coherent clusters. then, we measured the generalisation of the proposed approach via a cross-dataset evaluation by performing clustering on a novel dataset with unknown aspect categories. finally, we showed the benefit of this approach on the traditional stance classification task, along with a report on the thorough ablation study highlighting the impact of each model component on the clustering quality and the degree of disentanglement of the generated representations. our contributions can be summarized as follows: more recently, @xcite presented a sentence bottleneck autoencoder, called autobot, that learns a latent code by reconstructing the perturbated text. their model indicates the importance of topic labels as reconstruction objectives. disentangled latent representation earlier works explored disentangled representation to facilitate domain adaptation @xcite @xcite . in recent years, @xcite generated disentangled representations geared to transfer holistic style such as tone and theme in text generation. @xcite proposed the swapping autoencoder to separate texture encoding from structure vectors in image editing. the input images are formed in pairs to induce the model to discern the variation (e.g., structure) while retaining the common property (e.g., texture). however, recent studies show that disentanglement in the latent space is theoretically unachievable without access to some inductive bias @xcite . it is suggested that local isometry between variables of interest is sufficient to establish a connection between the observed variable and the latent variable @xcite , even with few annotations @xcite . this is in line with @xcite where contrastive pairs are leveraged for training, which illuminates our work to utilize labels and reconstruction of perturbed text to induce the disentanglement. text clustering the recent development in neural architectures has reshaped clustering practices @xcite . for example, @xcite leveraged transformer encoders for clustering over the user intents. several methods utilised plm embeddings to discover topics which were subsequently used for clustering news articles and product reviews @xcite . others exploited the neural components, i.e., the bilstm-cnn @xcite , the cnn-attention @xcite and the self-attention @xcite to offer endto-end clustering. zhang et al. (2021a) developed the supporting clustering with contrastive learning (sccl) model by augmenting the disparity between short text. a notable work is ds-clustering @xcite , which extracts aspect phrases first then clusters the aspect embeddings. outside of clustering methods, there is a surging interest in clustering-friendly representations @xcite . yet, few methods cluster documents along a particular axis or provide disentangled representations to cluster over a subspace. ## methodology we build our approach upon two vaccination opinion corpora @xcite . in both corpora, a small number of tweets are labelled, each of which is annotated with a stance label @xcite and a text span or an argumentative pattern denoting an aspect. for example, for the tweet, 'the pfizer vaccine is safe.', its stance label is 'provaccine' and the argumentative pattern is 'vaccine safety'. since vaccination opinions explode over time, supervised classifiers or aspect extractors would soon become outdated and fail to handle constantly evolving tweets. in an effort to mitigate this issue, we address the problem of vaccination opinion mining by learning disentangled stance and aspect vectors of tweets in order to cluster tweets along the aspect axis. our proposed model, called disentangled opinion clustering (doc), is shown in figure 1 . it is trained in two steps. in unsupervised learning (figure 1(a) ), a tweet is fed into an autoencoder with deberta as both the encoder and the decoder to learn the latent sentence vector z. here, each tweet is mapped to two embeddings, the context embedding u s which encodes the stance label information and the aspect embedding u a which captures the aspect information. under unsupervised learning, these two embeddings are not distinguished. together with the hidden representation of the input text, h, they are mapped to the latent sentence vector z by cross-attention. as the autoencoder can be trained on a large-scale unannotated tweets relating to vaccination, it is expected that z would capture the vaccine-related topics. then in the second step of supervised learning (figure 1(b) ), the deberta-based autoencoder is fine-tuned to learn the latent stance vector z s and the latent aspect vector z a using the tweet-level annotated stance label and aspect text span (or the argumentative pattern 'vaccine safety' in figure 1(b) ) as the inductive bias. here, the latent stance vector z s is derived from u s . it is expected that z s can be used to predict the stance label. on the other hand, the latent aspect vector z a is derived from u a only and it can be used to generate the sbertencoded aspect text span. both z s and z a , together with the hidden representation of the input text h, are used to reconstruct the original text through the deberta decoder. the training instances are organized in pairs since we use the idea of swapped autoencoder (shown in figure 1(c )) to swap the aspect embedding of one tweet with that of another if both discuss the same aspect. the resulting latent vector can still be used to reconstruct the original tweet. in what follows, we describe the two steps, unsupervised and supervised learning, in detail. unsupervised learning of sentence representation due to the versatility of plms, sentence representations are usually derived directly from contextualised representations generated by the plms. however, as has been previously discussed in @xcite , sentence representations derived in this way cannot guarantee reliable reconstruction of the input text. partly inspired by the use of autoencoder for sentence representation learning as in @xcite , we adopt the autoencoder sbert('vaccine safety') 'the pfizer is safe' z a a z a s z a c deberta encoder cross attention u a s u a a deberta decoder 'the pfizer vaccine is safe' n-mlp softmax stance label aspect span y ˆa 'the pfizer is safe' z deberta encoder cross attention us ua deberta decoder 'the pfizer vaccine is safe' 'mrna are poison' z b c z b a za a z b s zb c deberta encoder cross attention u b s u a a deberta decoder 'mrna vaccines are poison' sbert('vaccine safety') u b a (a) unsupervised learning (b) supervised learning (c) swapped autoencoder to enable a better disentanglement of z s and z a , for the two tweets discussing the same aspect but with different stance labels, tweet b's aspect embedding u b a is replaced by the tweet a's aspect embedding u a a . as the two tweets discuss the same aspect, their aspect embeddings are expected to be similar. as such, we can still reconstruct tweet b using the latent content vector z b c derived from the swapped aspect embedding. note that (b) and (c) are learned simultaneously. more specifically, we train the autoencoder on an unannotated twitter corpus with the masked token prediction as the training objective. the encoder applies the multi-head attention to clamp the hidden representations of the top layer of the pre-trained transformer. if we use h to denote the hidden representations, the multi-head attention can be expressed as: ## experiments datasets we conduct our experimental evaluation on two publicly available twitter datasets about the covid-19 vaccination: the covid-moral-foundation (cmf) @xcite and the vaccination attitude detection (vad) corpus @xcite . cmf is a tweet dataset focused on the covid-19 vaccine debates, where each tweet is assigned an argumentative pattern. vad consists of 8 aspect categories further refined by vaccine bands. similar to the argumentative pattern in the cmf dataset, each tweet is characterised by a text span indicating its aspect. the dataset statistics are reported in baselines we employ 5 baseline approaches: sbert foot_4 , autobot foot_5 , ds-clustering, vadet, and sccl foot_6 , of which sbert and autobot are out-ofthe-box sentence embedding generators. vadet is specialised to learn disentangled representations. however, it is noteworthy that even though it employs dec @xcite , the resulting representations are unsuitable for distance-based clustering. sccl performs joint representation learning and document clustering. ds-clustering is a pipeline approach that predicts a text span and employs sbert to generate an aspect embedding. for clustering-friendly representation learning methods, we examine their performance using k-means and k-medoids @xcite , and the agglomerative hierarchical clustering (ahc). the comparison involves three tasks: tweet clustering based on aspect categories (intra-and crossdatasets), and tweet-level stance classification. for stance classification, we employ roberta and de-berta, and use their averaged embeddings for clustering. evaluation metrics first, we use clustering accuracy (ca) and normalized mutual information (nmi) to evaluate the quality of clusters in line with @xcite . nmi is defined as @xmath0; ŷ) / h(y)+h(ŷ) , where i(y; ŷ) denotes the mutual information between the ground-truth labels and the predicted labels, h(•) denotes their entropy. then we employ bertscore @xcite to evaluate the performance of models in clustering in the absence of ground-truth cluster labels. bertscore is a successor of cosine similarity @xcite that measures the sentence distance by calculating the cross distance between their corresponding word embeddings. we follow @xcite to compute the averaged bertscore as ## conclusion in this work, we introduced doc, a disentangled opinion clustering model for vaccination opinion mining from social media. doc is able to disentangle users' stances from opinions via a disentangling attention mechanism and a swap-autoencoder. it was designed to process unseen aspect categories thanks to the clustering approach, leveraging clustering-friendly representations induced by outof-the-box sentence-bert encodings and the disentangling mechanisms. a thorough experimental assessment demonstrated the benefit of the disentangling mechanism on the quality of aspect-based clusters and the generalization capability across datasets with different aspect categories outperforming existing approaches in terms of generalisation and coherence of the generated clusters.