A Few Topical Tweets are Enough for Effective User Stance Detection (2021)

Abstract: User stance detection entails ascertaining the position of a user towards a target, such as an entity, topic, or claim. Recent work that employs unsupervised classification has shown that performing stance detection on vocal Twitter users, who have many tweets on a target, can be highly accurate (+98%). However, such methods perform poorly or fail completely for less vocal users, who may have authored only a few tweets about a target. In this paper, we tackle stance detection for such users using two approaches. In the first approach, we improve user-level stance detection by representing tweets using contextualized embeddings, which capture latent meanings of words in context. We show that this approach outperforms two strong baselines and achieves 89.6% accuracy and 91.3% macro F-measure on eight controversial topics. In the second approach, we expand the tweets of a given user using their Twitter timeline tweets, which may not be topically relevant, and then we perform unsupervised classification of the user, which entails clustering a user with other users in the training set. This approach achieves 95.6% accuracy and 93.1% macro F-measure.

## introduction stance detection entails identifying the position of a user towards a topic, an entity, or a claim @xcite . effective stance detection, particularly in the realm of social media, can be instrumental in gauging public opinion, identifying intersecting and diverging groups, and understanding issues of interest to different user communities @xcite . much recent works have explored varying stance detection methods including supervised, semi-supervised, and unsupervised user classification @xcite @xcite , and much of the work has focused on stance detection for twitter users. the different approaches have advantages and disadvantages. for example, supervised methods are simple to implement, but they require manually annotated training data and their accuracy varies widely based on classification features, classification techniques, and the number of training and test examples @xcite . though semisupervised and unsupervised methods typically use user interactions and often may yield perfect classification, they are effective in classifying highly vocal users with many topical tweets @xcite . most of these methods produce sub-optimal results for users who rarely express their opinion, and for whom we may only have one or two topically related tweets. though a single tweet might be explicitly clear, often it may lack sufficient context to determine the stance of the user. figure 1 show two tweets that pertain to the 2018 us midterm elections, where the first expresses a lucid pro-republican stance and the second could have been authored by a supporter of either the republican or democratic party. in this paper, we aim to effectively identify the stance of twitter users towards specific targets (entities or topics), where the users have mentioned the targets in only a few tweets (less than two tweets on average). to do so, we employ two approaches. in the first approach, we classify users based on their tweets that are represented using contextualized embeddings, which capture latent meanings of words in context. specifically, we use bert embeddings to represent tweets, and we fine tune the embeddings for every topic. we compare this approach to two strong baselines, namely using support vector machine (svm) classification and fasttext, which is a deep learning based classifier. in the second approach, we expand the tweets of a given user using their twitter timeline tweets, and then we use the additional tweets, which would typically be not relevant to the topic at hand, to perform unsupervised classification of the user by clustering him/her with the users in the training set. using such expansion allows us to make use of user homophily, which is manifested in the echo chambers that form on twitter, where users with similar views tend to retweet similar accounts beyond the topic at hand. to test our approaches, we use a dataset containing tweets on 8 polarized us-centric topics. we also examine the effect of expansion when using svm, fasttext, and contextualized embeddings. for testing, we randomly selected 100 users for each topic that have less than 5 topical tweets, and we manually labeled them for stance. to construct the training set, we used unsupervised stance detection to automatically label the 5,000 most active users per topic, and for every topic we used a balanced set of 500 users per stance as our training set @xcite . since the approaches rely on different features and utilize different classification techniques, we indicate which approach works best under different conditions. the contributions of this paper are as follows: ## related work over the last few years, much research has focused on user stance detection. the goal of stance detection is to ascertain the positions of users towards some target such as a topic, person, or claim @xcite @xcite @xcite @xcite @xcite @xcite @xcite . while stance may easily be detected by humans, machine learning models often fall short, particularly for users who talk about a target sparingly. several studies have focused on modeling stance by introducing different features ranging from linguistic and structural features @xcite to network interactions and profile information @xcite @xcite . much work on stance detection involved using supervised and semi-supervised classification methods. one of the major downsides of both classification methods is the need for a seed list of manually labeled users, which is time consuming and requires topic expertise. supervised learning is sensitive to the classification features, the size of the training sets, the number of available tweets for users in the test set, and the classification algorithm @xcite . some common classification features include: lexical, syntactic, and semantics feature; network features such as retweeted accounts and user mentions; content features such as words and hashtags; and user profile information such as name and location @xcite @xcite . some commonly used classification algorithms include svms and deep learning classification @xcite . @xcite presented a neural network model for stance classification by augmenting bert representations with a novel consistency constraint to determine stance with respect to both a claim and perspective. we extend their work in two ways, namely: we drop the need to have a claim and perspective, and we couple bert supervised classification with unsupervised classification to effectively tag vocal and non-vocal users. semi-supervised methods such as label propagation @xcite @xcite often rely on two users retweeting identical accounts or tweets to propagate a label of one user to another. though such typically achieves high precision (often above 95%) @xcite , it is generally successful in tagging vocal users with strong opinions. recently, @xcite have introduced a highly effective unsupervised method for predicting the stance of prolific twitter users towards controversial topics. by projecting users onto a low-dimensional space and then clustering them allows for clear separation between vocal users with respect to their stance @xcite . this method confers two main advantages over previous methods, namely: it does not require any initial manual labeling, and classification accuracy is nearly perfect. however, it is successful in labeling vocal users only and fails on users with very few topical tweets. we extend prior work on unsupervised stance detection to effectively classify both prolific and non-prolific users in a holistic way by aggregating both supervised and unsupervised methods. further, we extend prior deep-learning based supervised classification to use contextual embeddings that capture syntactic and semantic features of words in context. we are framing the problem as user-based classification, which is common in the computational social science community, as opposed to tweetbased classification, which is common in the nlp community @xcite . this is motivated by two aspects, namely: 1) tweets often don't provide sufficient context for proper annotation; and 2) users have durable stances over time. for example, if someone says "most important election in history! vote!", it is nearly impossible to know if the author's position without context. topics our dataset includes tweets on eight polarizing topics that are us-centric, which were gra-ciously provided to us by @xcite . contextualized embeddings over the last several years, pre-trained embedding @xcite have helped achieve significant improvements in a wide range of classification tasks in natural language processing. representing words as vectors in a low-dimensional continuous space and then using them for downstream tasks lowered the need for extensive manual feature engineering. however, these pre-trained vectors are static and fail to handle polysemous words, where different instances of a word have to share the same representation regardless of context. more recently, different deep neural language models have been introduced to create contextualized word representations that can cope with the issue of polysemy and the context-dependent nature of words. models such as openai gpt @xcite , elmo @xcite , bert (bidirectional encoder representations from transformers) @xcite , and umlfit @xcite , to name a few, have achieved groundbreaking results in many nlp classification and language understanding tasks. for this paper, we use bert base-multilingual foot_1 (referred to hereafter simply as bert), which we fine-tune for stance detection, as this eliminates the need for heavily engineered task-specific architectures. bert is pre-trained on wikipedia text from 104 languages and comes with hundreds of millions of parameters. it contains an encoder with 12 transformer blocks, hidden size of 768, and 12 self-attention heads. as shown in fig. 2 , we fine-tuned bert by adding a fully-connected dense layer followed by a softmax output layer, minimizing the binary cross-entropy loss function for the training data. for all experiments, we used huggingface foot_2 transformer implementation with pytorch foot_3 as it provides pre-trained weights and vocabulary. as for features, we used all the words in tweets that were preprocessed in the manner described earlier for svm and fasttext. similar to fasttext, we performed tweet-level clas-sification, and we used the average softmax output scores per class across all tweets for a user to assign a label to a test user. unsupervised classification for unsupervised classification, we used the same unsupervised classification method described earlier, which we used to prepare the training set. specifically, we constructed a feature vector for each test user based on the accounts he/she retweeted, computed its similarity to all users in the training set, projected all the users in the training along with the test user into a lower dimensional space using umap, and lastly clustered the users using mean shift. we then labeled the test user using the majority label of the cluster in which the user appeared. ## experiments we split users in our test set on the basis of whether we were able to crawl their timelines or not. set a includes users for which we were not able to obtain their timeline tweets. set b includes users for which we were able to collect their timeline tweets. we separated between them, because set b would allow us to compare between setups that use timeline tweets with those that do not on identical users. for set a, we always trained on the training users with their on-topic tweets and the tested on the test users, who typically had less than 2 tweets on average. we used four different classification setups, namely using fasttext, svm with retweeted accounts as features (svm rt ), svm with all words as features (svm t ext ), and finetuned bert embeddings with a dense neural layer and softmax output (bert). we experimented with using the unsupervised method on set a, but the unsupervised algorithm was not able to assign any test user to a cluster, mostly because the number of tweets and subsequently retweeted users per test user were too few. for set b, we experimented with the same classifiers using four different conditions, namely: not expanding either training or test sets with users' timeline tweets; expanding the test set only; expanding the training set only; and expanding both the training and test sets. ## results and discussion for all experiments, we report on per topic accuracy (a) and macro precision (p), recall (r), and f-measure (f) across stances on a topic. we computed the standard deviation (sd) of all our measures across topics for every setup. lower sd coupled with high a and f is desirable as they indicate the setup produces consistently high results across topics. unsupervised classification yielded the lowest sd values and highest overall score. bert and fasttext with no expansion and svm rt with expanded test set had slightly higher sd. thus, if we are able to scrape a user's timeline tweets, it is advantageous to use a method that relies on which accounts a user retweets, with unsupervised classification producing the best results. as we will show in the error analysis, the success of ## conclusion in this paper, we presented two methods for classifying users according to their stance towards a target. the first utilizes contextualized embeddings to represent tweets and then uses deep neural network for classification. this approach led to results that outperform two strong baselines. the second utilizes additional tweets from users' timelines to cluster test users with other users with known labels in an unsupervised manner. the first method yielded the best results when timeline tweets were not available, while the second yielded even better results overall. given the overall setup described in the paper, where the training data was obtained using unsupervised user classification, we can automatically label the most active users with nearly perfect accuracy, and we can label users with only few topical tweets with high accuracy, often above 95% when we can obtain their timeline tweets. for future work, we plan to explore the effectiveness of cross topic classification, where training and testing are done on different topics. perhaps, we can build unified models that could be used across multiple topics for a given a population of twitter users (ex. users who are interested in us politics).