Classification without (Proper) Representation: Political Heterogeneity in Social Media and Its Implications for Classification and Behavioral Analysis (2022)

Abstract: Reddit is home to a broad spectrum of political activity, and users signal their political affiliations in multiple ways—from self-declarations to community participation. Frequently, computational studies have treated political users as a single bloc, both in developing models to infer political leaning and in studying political behavior. Here, we test this assumption of political users and show that commonly-used political-inference models do not generalize, indicating heterogeneous types of political users. The models remain imprecise at best for most users, regardless of which sources of data or methods are used. Across a 14-year longitudinal analysis, we demonstrate that the choice in definition of a political user has significant implications for behavioral analysis. Controlling for multiple factors, political users are more toxic on the platform and inter-party interactions are even more toxic—but not all political users behave this way. Last, we identify a subset of political users who repeatedly flip affiliations, showing that these users are the most controversial of all, acting as provocateurs by more frequently bringing up politics, and are more likely to be banned, suspended, or deleted.

## introduction individuals readily engage in political behavior online, sharing content and forming communities with like-minded individuals. scholars study these active political communities to understand partisanship @xcite , polarization @xcite , and voting behaviors @xcite . many computational studies of political behavior in social media have the underlying assumption that political leanings can be reliably identified. prior work has shown that partisan leaning can be inferred from a diverse set of behavioral characteristics such as text @xcite , social networks @xcite , and even community participation @xcite . yet, inferring political leaning is known to be a challenging problem @xcite , in particular for centrist or apolitical users who infrequently express political beliefs. further, inference models typically use a single source of information on a user's political affiliation without examining whether this source generalizes to all types of users. this methodology fails to account for the disparate types of political users and introduces sampling bias downstream. here, we re-examine inferring political behavior for these diverse groups in a unified setting to understand the consequences our data have on results. this paper tests what effect current assumptions of social media users' political affiliations have on our ability to model political users and their behaviors. the first part of the paper tests how different definitions of political users generalize to other users' behaviors and to inferring political leaning. using 574k political users on reddit, we show that the common definitions of a political user (e.g., those making self-declarations of affiliation) result in behaviorally diverse types of users. moreover, we demonstrate that multiple computational approaches for political inference do not generalize across these political user types; our results show that political inference on reddit is challenging, with our best model for inference only attaining a 0.60 auc score across all users. the second part of the paper tests whether the choice in which type of political user influences the outcomes of political analyses. we show that controlling for multiple factors, political users are generally more toxic on the platform and that cross-affiliation interactions are even more toxicwith liberal-to-conservative interactions being most toxic. however, not all types of political users are equally toxic, highlighting the importance of how studies define political users. in addition, we identify a small set of users who near-simultaneously declare differing political affiliations. these users act as provocateurs and have substantially more controversial comments-with the most active eventually being banned. together, our study has substantial implications for future work on political behavior on reddit and highlights the need to account for different types of political users. ## political affiliation online online communities are active spaces for political discussions and cross-community engagement. researchers have examined how these political spaces, and the users therein, influence real-life politics @xcite , forecast future political outcomes, @xcite , increase political engagement offline @xcite and even polarize opinions @xcite . such research depends on knowing the political affiliation of users. people online may express their political affiliations explicitly or implicitly, and not all users reveal their affiliations @xcite . this lack of data potentially limits large studies of political engagement. as a result, substantial work has focused on inferring affiliation to increase the data representativeness (e.g., @xcite @xcite @xcite . however, political inference is known to be challenging, and prior work has shown methods often fail to generalize to users outside the narrow range of political orientation on which they were trained @xcite . moreover, the majority of work uses only a single source of ground truth-when multiple are available-without testing the implications of which type of user makes the political declaration and whether those users are representative at large. this study tests this underlying assumption of generalizability of how political users are defined and what effect this has on affiliation inference and behavioral studies of political users. ## identifying political affiliation individuals signal their political beliefs in multiple ways from self-declarations to participation in partisan communities. these different sources of information offer complementary ways of recognizing beliefs, and defining who exactly is a "political user." prior works have varied significantly in which of these signals they use (e.g., @xcite . here we define political affiliation and describe different sources of political identification. our data collection matches prior work, and we rely on previous definitions of political affiliation to build our set of political users. ## characterizing political behavior do users who declare their political beliefs in different ways also behave differently? here, we test for behavioral differences between user categories. behavior by data source we analyzed general behavioral differences between political users and non-political users using a random sample of 10k users from each category and 10k non-political users. for each category, we measure (i) how old their accounts are as the time between first and last comment and (ii) diversity in community participation as entropy over subreddits. substantial variation was seen across the groups. users with no declared affiliation had accounts nearly twice as old (µ=94 months) as political users (µ=46), and for every political data source, conservatives have a shorter lifespan of activity. the median longevity for conservative users is a full year less than their liberal counterparts. conservatives in the flair dataset have the shortest overall lifespan with a median of 31 months. as reddit's user base has grown substantially since its beginningparticularly with an influx of political users around the 2016 u.s. election-our results point to the need to recognize political and non-political users as heterogeneous groups. political users varied in how widely they comment across communities, with users who selfdeclare and those with flairs participating in more communities on average. figure 1 shows the mean entropy for user type, revealing users in two groups participate more broadly than those whose affiliation is derived from participating in partisan communities (@xmath0.01). the entropy is calculated from the probability that user u j comments in a subreddit s @xmath1: conservatives vs liberals conservatives and liberals are known to operate in different bubbles online @xcite . here, we test whether the different groups within an affiliation have separate bubbles themselves. political users are represented by their commenting frequencies across subreddits. pca is applied to identify latent variations in where users are active. figure 2a shows the t-sne projection of these political users colored by affiliation with shapes for each user type; closeness in this plot indicates users are active in the same communities. this projection shows three trends. first, as expected, some conservative and liberal users participate in bubble-like spaces with users of primarily one affiliation. second, surprisingly, some clusters exhibit strongly-mixed affiliation, indicating that reddit is not entirely polarized and some users do regularly interact across affiliations. third, some politically affiliated clusters are primarily made of one user type as shown in figure 2b which plots using points colored by user type. this result suggests that several micro-bubbles exist where users may not interact with others of their affiliation. as a result, computational studies using only one source of data may incorrectly estimate how information spreads between users or the norms of political users in a community. see appendix b.1 for pca and t-sne hyperparameters) ## inferring political affiliation multiple methods have been proposed for inferring political affiliation. however, these methods have typically used only a single source of information as ground truth (e.g., community membership). given the behavioral differences between observed users from different sources of information, we test how well a broad set of approaches identifies political affiliation and to what degree does an approach and source of ground truth generalize to inferring the affiliation for other types of users. additional details on the hyperparameter settings for each model are detailed in appendix §b. username classifier usernames can reveal aspects of identity @xcite , e.g., hillary4prez reveals a liberal leaning. to predict affiliation from names, we follow @xcite and train a bidirectional character-based lstm. text classifier some topics are politically oriented and can potentially reveal a user's leaning, e.g., discussing interests in gun rights. to infer affiliation from such statements, we train a roberta @xcite model over comments made from each user, excluding any statements they make that explicitly self-identify their affiliation. the model predicts each comment, and we aggregate the model outputs by taking the mean of predictions for selected comments associated with a user as the final label. ## political interactions and engagement political discussions are known to be heated @xcite , and online discussions of political topics are more uncivil and aggressive than nonpolitical topics @xcite . in part, political topics have become increasingly moralized @xcite , where discussions are more connected to a person's identity. here, we examine the interactions between political users to probe the mechanisms behind this toxicity. reddit allows communities to discuss political topics with like-minded individuals, but also allows common spaces for both political and non-political topics for all @xcite . as a result, we test whether these discussions become more uncivil due to political persons or the topic itself. further, given the clear differences seen between our groups of political users, we test whether these users behave differently to test for potential confounds from only studying one group. ## two-faced actors our data identified a small percent of political users who declare different political affiliations within a short period. given the rise in trolls and other malicious actors on social media @xcite , we ask whether these users, who we refer to as two-faced actors, behave differently than other types of political users. ## changing political beliefs in american politics, political beliefs have shifted closer to closer-held ideological beliefs @xcite . however, some individuals do change affiliation. are users of one type more likely to switch parties? here, we test whether affiliation changes can be predicted from prior behavior. analysis to identify changing behavior, we use a time constraint that requires the change in political affiliation to occur at least one year following the user's original political declaration. a total of 2,076 affiliation-changing users were identified through this process. the changing of affiliation was split roughly evenly, with 56% of flips going from conservative to liberal, and 44% going from liberal to conservative. flips are not unique to one type of user, but users do differ in their probability of flipping, with self-declared and flair users being an order of magnitude more likely to flip (2.8% and 1.6%, relatively) than community-based-users (0.2%). for users who changed affiliations, only 17.5% of their accounts have been suspended or deleted, which is significantly lower than the twofaced actor rate and lower than the mean rate for non-political user (21.0%), suggesting these affiliation changes are likely done in good faith. experimental setup to control for confounds from behavioral differences, we created our dataset using coarsened matching to pair users who change their affiliations with political users who do not. two users are paired by having the same initial politics, closest comment count, and activity lifespan. for each matched pair of users, we collect six months of features prior to the change of political affiliation. the feature set includes the data source of the users, their original political declarations, and participation in popular and political subreddits. a complete list of the model features can be found in appendix §b.3. we train a logistic regression model to predict whether a user will change their political affiliation. we evaluate separate models for conservatives who became liberals, liberals who become conservatives, and a combined affiliation-independent model to analyze components of change regardless of party. ## conclusion social media is rife with political activity and research on these political spaces depends on accurate measurement of political users. we examine political users on reddit and show that the choice in how political users are defined-the evidence used to establish ground truth-has substantial consequences for downstream models and analyses. in particular, user groups from different definitions behave differently ( §4 and §6) and models trained on one type of user do not necessarily generalize to other groups ( §3). in three studies of political users, we show that (i) political users themselves drive hostility on the platform-with conservative users being the recipients of more toxicity, (ii) a small-but-very-active group of provocateurs simultaneously declare different affiliations and are a notable source of toxicity and controversiality on the platform, and (iii) changing political affiliation can be predicted, but performance varies considerably by user type. across all three studies, we show that the type of political user matters, with different types having substantially different behavior. models, data, and code for this study will be released at https://github.com/davidjurgens/ reddit-political-affiliation .