Predicting the Topical Stance and Political Leaning of Media using Tweets (2020)

Abstract: Discovering the stances of media outlets and influential people on current, debatable topics is important for social statisticians and policy makers. Many supervised solutions exist for determining viewpoints, but manually annotating training data is costly. In this paper, we propose a cascaded method that uses unsupervised learning to ascertain the stance of Twitter users with respect to a polarizing topic by leveraging their retweet behavior; then, it uses supervised learning based on user labels to characterize both the general political leaning of online media and of popular Twitter users, as well as their stance with respect to the target polarizing topic. We evaluate the model by comparing its predictions to gold labels from the Media Bias/Fact Check website, achieving 82.6% accuracy.

## introduction online media and popular twitter users, which we will collectively refer to as influencers, often express overt political leanings, which can be gleaned from their positions on a variety of political and cultural issues. determining their leaning can be done through the analysis of their writing, which includes the identification of terms that are indicative of stance @xcite . performing such analysis automatically can be done using supervised classification, which in turn would require manually labeled data @xcite @xcite . alternatively, leanings can be inferred based on which people share the content (blogs, tweets, posts, etc.) on social media, as social media users are more likely to share content that originates from sources that generally agree with their positions @xcite @xcite . here, we make use of this observation to characterize influencers, based on the stances of the twitter users that share their content. ascertaining the stances of users, also known as stance detection, involves identifying the position of a user with respect to a topic, an entity, or a claim @xcite . for example, on the topic of abortion in usa, the stances of left-vs. right-leaning users would typically be "pro-choice" vs. "pro-life", respectively. in this paper, we propose to apply unsupervised stance detection to automatically tag a large number of twitter users with their positions on specific topics @xcite . the tagging identifies clusters of vocal users based on the accounts that they retweet. although the method we use may yield more than two clusters, we retain the two largest ones, which typically include the overwhelming majority of users, and we ignore the rest. then, we train a classifier that predicts which cluster a user belongs to, in order to expand our clusters. once we have increased the number of users in our sets, we determine which sources are most strongly associated with each group based on sharing by each group. we apply this methodology to determine the positions of influencers and of media on eight polarizing topics along with their overall leaning: left, center or right. in doing so, we can also observe the sharing behavior of right-and leftleaning users, and we can correlate their behavior with the credibility of the sources. further, given the user stances for these eight topics, we train a supervised classifier to predict the overall bias of sources using a variety of features, including the so-called valence @xcite , graph embeddings, and contextual embeddings. using a combination of these features, our classifier is able to predict the bias of sources with 82.6% accuracy, with valence being the most effective feature. figure 1 outlines our overall methodology. our contributions are as follows: the remainder of this paper is organized as follows: section 2 discusses related work. section 3 describes the process of data collection. section 4 presents our method for user stance detection. section 5 describes how we characterize the influencers. section 6 discusses our experiments in media bias prediction. finally, section 7 concludes and points to possible directions for future work. ## related work recent work that attempted to characterize the stance and the ideological leaning of media and twitter users relied on the observation that users tend to retweet content that is consistent with their world view. this stems from selective exposure, which is a cognitive bias that leads people to avoid the cognitive overload from exposure to opposing views as well as the cognitive dissonance in which people are forced to reconcile between their views and opposing views @xcite . 2013 ) studied retweet behavior to infer the ideological leanings of online media sources and popular twitter accounts. @xcite proposed a statistical model based on the follower relationships to media sources and twitter personalities in order to estimate their ideological leaning. as for individual users, much recent work focused on stance detection to determine a person's position on a topic including the deduction of political preferences @xcite @xcite @xcite @xcite @xcite @xcite @xcite . user stance classification is aided by the tendency of users to form so-called "echo chambers", where they engage with like-minded users @xcite , and the tendency of users' beliefs to be persistent over time @xcite @xcite . studies have examined the effectiveness of different features for stance detection, including textual features such as word n-grams and hashtags, network interactions such as retweeted accounts and mentions, and profile information such as user location @xcite @xcite . network interaction features were shown to yield better results compared to using textual features @xcite . @xcite leveraged both user interactions and textual information when modeling stance and disagreement, using a probabilistic programming system that allows models to be specified using a declarative language. trabelsi and zaïane (2018) described an unsupervised stance detection method that determines the viewpoints of comments and of their authors. it analyzes online forum discussion threads, and therefore assumes a certain structure of the posts. it also assumes that users tend to reply to each others' comments when they are in disagreement, whereas we assume the opposite in this paper. their model leverages the posts' contents, whereas we only use the retweet behavior of users. many methods involving supervised learning were proposed for stance detection. such methods require the availability of an initial set of labeled users, and they use some of the aforementioned features for classification @xcite @xcite . such classification can label users with precision typically ranging between 70% and 90% @xcite . label propagation is a semisupervised method that starts with a seed list of labeled users and propagates the labels to other users who are similar based on the accounts they follow or retweet @xcite @xcite . while label propagation may label users with high precision (often above 95%), it is biased towards users with more extreme views; moreover, careful choice of thresholds is often required, and post-checks are needed to ensure quality. @xcite and more recently darwish et al. (2020) used unsupervised stance detection, where users are mapped into a lower dimensional space based on user-user similarity, and then clustered to find core sets of users representing different stances. this was shown to be highly effective with nearly perfect clustering accuracy for polarizing topics, and it requires no manual labeling of users. here, we use the same idea, but we combine it with supervised classification based on retweets in order to increase the number of labeled users @xcite . other methods for user stance detection include collective classification @xcite , where users in a network are jointly labeled and classification in a low-dimensional user-space @xcite . as for predicting political leaning or sentiment, this problem was studied previously as a supervised learning problem, where a classifier learns from a set of manually labeled tweets @xcite @xcite . similarly, @xcite predicted twitter users' political affiliation (being republican or democratic), using their network connections and textual information, relying on user-level annotations. ## data collection we obtained data on eight topics that are considered polarizing in the usa @xcite , shown in they include a mix of long-standing issues such as racism and gun control, temporal issues such as the nomination of judge brett kavanaugh to the us supreme court and representative ilhan omar's polarizing remarks, as well as non-political issues such as the potential dangers of vaccines. further, though long-standing issues typically show rightleft polarization, stances towards omar's remarks are not as clear, with divisions on the left as well. since we are interested in us users, we filtered some tweets to retain such by users who have stated that their location was usa. we used a gazetteer that included words that indicate usa as a country (e.g., america, us), as well as state names and their abbreviations (e.g., maryland, md). other data that we used in our experiments is a collection of articles that were cited by users from the tweets collection and that originate from media, whose bias is known, i.e., is discussed on the media bias/fact check website. ## user stance detection in order to analyze the stance of influencers on a given topic, we first find the stances of twitter users, and then we project them to the influencers that the users cite. a central (initial) assumption here is that if a user includes a link to some article in their tweet, they are more likely to agree or endorse the article's message. similarly, when a user retweets a tweet verbatim without adding any comments, they are more likely to agree with that tweet. we label a large number of users with their stance for each topic using a two-step approach, namely projection and clustering and supervised classification. for the projection and clustering step, we identify clusters of core vocal users using the unsupervised method described in @xcite . in this step, users are mapped to a lower dimensional space based on their similarity, and then they are clustered. after performing this unsupervised learning step, we train a supervised classifier using the two largest identified clusters in order to tag many more users. for that, we use fasttext, a deep neural network text classifier, that has been shown to be effective for various text classification tasks @xcite . once we have expanded our sets of labeled users, we identify influencers that are most closely associated with each group using a modified version of the so-called valence score, which varies in value between -1 and 1. if an influencer is being cited evenly between the groups, then it would be assigned a valence score close to zero. conversely, if one group disproportionately cites an influencer compared to another group, then it would be assigned a score closer to -1 or 1. we perform these steps for each of the given topics, and finally we summarize the stances across all topics. below, we explain each of these steps in more detail. ## characterizing the influencers we use valence to characterize the leaning of all cited influencers for each of the topics. of the 5,406 unique media sources that have been cited in tweets across all topics, 806 have known political bias from mediabiasfactcheck. com. figure 3 shows the confusion matrix between our valence categories and the goold labels from mediabiasfactcheck.com. we notice that many of the media that have a negative valence score (categoriesand --) are classified on the right side of the political spectrum by mediabiasfactcheck.com, while most media with positive scores (categories + and ++) are classified as slightly left-leaning. although there are almost no extreme-left cases, there is a correlation between bias and our valence score. mediabiasfactcheck.com seems to rarely categorize media sources as "extreme-left". this could be a reflection of reality or it might imply that mediabiasfactcheck.com has an inherent bias. we also computed the valence scores for the top-200 retweeted accounts, and we assigned each account a valence category based on the score. independently, we asked a person who is well-versed with us politics to label all the accounts as left, center, or right. when labeling accounts, right-leaning include those expressing support for trump, the republican party, and gun rights, opposition to abortion, and disdain for democrats. as for left-leaning accounts, they include those attacking trump and the republicans, and expressing support for the democratic party and for liberal social positions. if the retweeted account happens to be a media source, we used mediabiasfactcheck.com. it is noteworthy that all top-200 retweeted accounts have extreme valence categories on average across all topics. their average valence scores, with one exception, appear between -0.6 and -1.00 for right, and between 0.6 and 1 for left (see figure 4 ). of those manually and independently tagged accounts, all that were tagged as left-leaning have a strong positive valence score and all that were tagged as right-leaning have a strong negative valence score. only two accounts were manually labeled as center, namely reuters and cspan, which is a us channel that broadcasts federal government proceedings, and they had valence scores of 0.55 and 0.28, respectively. though their absolute values are lower than those of all other sources, they are mapped to the + valence category. medium factuality bias average climate change gun control ilhan immigration midterm police & racism kavanaugh vaccine thehill.com h l-c + + + 0 ++ + + + + ++ ++ theguardian.com h l-c ++ ++ ++ ++ ++ ++ ++ ++ ++ ++ ++ washingtonpost.com h l-c ++ ++ ++ ++ ++ ++ ++ ++ ++ ++ ++ breitbart.com ## predicting media bias given the stances of users on the aforementioned eight topics, we leverage this information to predict media bias. specifically, we describe in this section how we make use of the valence scores, as well as other features, namely graph and contextualized text embeddings, to train supervised classifiers for this purpose. valence scores. we use valence scores in two ways. first, we average the corresponding valence across the different polarizing topics to obtain an average valence score for a given target news medium. this is an unsupervised method for computing polarity. second, we train a logistic regression classifier that uses the calculated valence scores as features and annotations from mediabiasfactcheck.com as gold target labels in order to predict the general political leaning of a target news medium. we merged "left" and "extreme left", and similarly we merged "right" and "extreme right". we discarded media labeled as being "leftcenter" and "right-center". each news medium was represented by an 8-dimensional vector containing the valence scores for the above topics. in the experiments, we used the lbfgs solver and @xmath0.1. we used two measures to evaluate its performance, namely accuracy and mean absolute error (mae). the latter is calculated by considering the different classes as ordered and equally distant from each other, i.e., if the model predicts right and the true label is left, this amounts to an error equal to 2. climate change gun control ilhan omar immigration theguardian.com h l-c thehill.com h l-c washingtonpost.com h l-c theguardian.com h l-c washingtonpost.com h l-c cnn.com m l theguardian.com h l-c washingtonpost.com h l-c independent.co.uk h l-c nytimes.com h l-c mondoweiss.net h l cnn.com m l wef.ch npr.org vh l-c thinkprogress.org m l huffingtonpost.com h l vox.com h l washingtonpost.com h l-c haaretz.com h l-c npr.org vh l-c nytimes.com h l-c politico.com h l-c nytimes.com h l-c thehill.com h l-c bbc.com h l-c usatoday.com h l-c thehill.com h l-c nytimes.com h l-c cnn.com m l msn.com h l-c politico.com h l-c reuters.com vh c reuters.com vh c bbc.com h l-c cnn.com m l politico.com h l-c bloomberg.com h l-c cnbc.com h l-c apple.news usatoday.com h l-c thehill.com h l-c apple.news mediaite.com h l apple.news apple.news sun-sentinel.com h r-c usatoday.com h l-c msn.com h l-c npr.org vh l-c nypost.com m r-c yahoo.com m l-c pscp.tv seattletimes.com h l-c dailymail.co.uk vl r timesofisrael.com h l-c whitehouse.gov m r newsweek.com m l mailchi.mp theatlantic.com h l-c texastribune.org h c change.org h l washingtontimes.com h r-c nypost.com m r-c dailymail.co.uk vl r latimes.com h l-c breaking911.com vl jpost.com h c nypost.com m r-c dailymail.co.uk vl r chicagotribune.com h r-c dailymail.co.uk vl r zerohedge.com m climatechangedispatch.com rt.com m r-c algemeiner.com h r-c ir.shareaholic.com cnbc.com h l-c forbes.com m r-c startribune graph embeddings. we further use graph embeddings, generated by building a user-to-hashtag graph (u2h) and a user-to-mention (u2m) graph and then running node2vec on both @xcite , producing two types of graph embeddings. when using graph embeddings, we got worse results compared to our previous setup with valence scores (see tweets. we also experimented with bert-base. we used the text of the tweets that cite the media we are classifying. for classification, we fed bert representations of tweets to a dense layer with softmax output to fine-tune it with the textual contents of the tweets. we trained at the tweet level, and we averaged the scores (from softmax) for all tweets from the same news medium to obtain an overall label for that news medium. the accuracy is much lower than for the valence scores: 64.0% accuracy vs. 75.2% for supervised and 68.0% for unsupervised. article titles and text. using the bert setup for tweets, we used the titles and the full text of up to 100 articles from each of the target media. when using the full text of articles, we balanced the number of articles per news medium. we trained two separate bert models, one on the titles and another one on the full text (content). both models did worse than using valence alone, but the combination improved over valence only. system combination. we combined different setups including using all the aforementioned models in combination. using graph embeddings (graphh + graphm) with bert embeddings (tweet+title+content) and valence yielded the best results with accuracy of 82.6% and mae of .206. if we remove valence from the combination, the accuracy drops by 4.5% while mae jumps by .078, absolute. this suggests that valence is a very effective feature that captures important information, complementary to what can be modeled using graph and contextualized text embeddings. ## conclusion and future work we have presented a method for predicting the general political leaning of media sources and popular twitter users, as well as their stances on specific polarizing topics. our method uses retweeted accounts, and a combination of dimensionality reduction and clustering algorithms, namely umap and mean shift, in order to produce sets of users that have opposing opinions on specific topics. next, we expand the discovered sets using supervised learning that is trained on the automatically discovered user clusters. we are able to automatically tag large sets of users according to their stance of preset topics. users' stances are then projected to the influencers that are being cited in the tweets for each of the topics using the so-called valence score. the projection allows us to tag a large number of influencers with their stances on specific issues and with their political leaning in general (i.e., left vs. right) with high accuracy and with minimal human effort. the main advantage of our method is that it does not require manual labeling of entity stances, which requires both topical expertise and time. we also investigated the quality of the valence features, and we found that valence scores help to predict media bias with high accuracy. in future work, we plan to increase the number of topics that we use to characterize media. ideally, we would like to automatically identify such polarizing topics. doing so would enable us to easily retarget this work to new countries and languages.