OK Boomer: Probing the socio-demographic Divide in Echo Chambers (2022)

Abstract: Social media platforms such as Twitter or Reddit have become an integral part in political opinion formation and discussions, accompanied by potential echo chamber forming. In this paper, we examine the relationships between the interaction patterns, the opinion polarity, and the socio-demographic characteristics in discussion communities on Reddit. On a dataset of over 2 million posts coming from over 20k users, we combine network community detection algorithms, reliable stance polarity annotations, and NLP-based socio-demographic estimations, to identify echo chambers and understand their properties at scale. We show that the separability of the interaction communities is more strongly correlated to the relative socio-demographic divide, rather than the stance polarity gap size. We further demonstrate that the socio-demographic classifiers have a strong topical bias and should be used with caution, merely for the relative community difference comparisons within a topic, rather than for any absolute labeling.

## introduction social media platforms such as twitter or reddit have become an integral part in political opinion formation and discussions, as users exchange opinions on numerous polarising topics such as gun control, abortion or healthcare. this process is accompanied by the forming of echo chambers, i.e. clusters formed by users with a homogeneous content production and diffusion @xcite , where users mostly see posts reinforcing their preexisting belief @xcite . at the same time, humans tend to be homophile in their social connections and interactions in general, implying that socio-demographic simi-larities in categories such as age, gender, ethnicity, religion or political ideology significantly increase the chance of a connection between two individuals in social networks @xcite @xcite . so while we do know that similarities foster connections and common identities @xcite as well as that online communities can become echo chambers, by combining network-based community modeling, stance annotations, and socio-demographic projections from natural language of self-identified authors, it is possible to coarsely estimate the extent to which these phenomena inter-play such that this sociodemographic clustering is intensified in online echo chambers. based on that, we explore the following hypotheses in this paper: (i) key societal topics on reddit shape network interaction communities indicating the echo chamber phenomenon, (ii) stance polarity mean values are further apart in more separated network communities, (iii) a distinct sociodemographic divide exists between groups of interacting users with diverse stance polarities showing echo chamber characteristics, (iv) automated sociodemographic profiling tools suffer from a strong topical bias, which hinders their ability to characterize the communities. this paper provides the following contributions: • we create a reddit dataset of over 20k users (over 2m posts) within 8 current societal topics (sec. 3), aligned with manual stance polarity annotations of 640 users (sec. 4). ## related work recent works have either studied social-media data with regards to their graph-theoretical properties to detect echo-chamber-like phenomena from the user interactions @xcite @xcite @xcite or estimated sociodemographic properties of social media users with nlp methods in isolation @xcite @xcite . however, the combination of these two procedures in order to study a potential socio-demographic divide in such user groups at scale has, to the best of our knowledge, not been investigated so far. in the broader context of analysing the political orientation of users in combination with their demographics, @xcite studies how twitter users cluster with respect to different political leanings and shows that women tend to be on average slightly more liberal than men. a similar study demonstrated that there are differences in the average political leaning depending on gender, age, marital status and possession of a college degree @xcite , and observed that stronger ties between friends lead to a stronger correlation between their ideologies, which inspired us to the hypotheses explored in this paper. on a similar note, @xcite showed that mutual @-connections are more likely to appear between same-gender individuals. comparable clustering effects were found for age as well as ideology @xcite . furthermore, @xcite studied the relationship between echo chambers concerning the brexit referendum on twitter and the geographic lo-cation of its members, while @xcite found clear differences in the predicted stance towards donald trump between users from different us states, both embodying the idea of extracting social media stance-wise user groups and analysing their characteristics. regarding content-based models for the prediction of stance and socio-demographic properties, @xcite studied discriminating tokens in the joined prediction of gender and stance towards abortion, finding that these correlate to the two labels differently, hinting towards our hypothesized topical bias for tokens that correlate more with stance than gender. in the proposed work, we combine the users' political orientation, their estimated sociodemographic properties and their social media network, while previous works combine only a subset of these concepts, as shown in ## dataset characteristics we used the api of reddit to build our dataset. we chose reddit as our source of data since it provides (i) rich content, due to the fact that there is no word limit, and (ii) a clear relationship between the text and the target topic, since users post within a subreddit. previous work @xcite showed that the controversiality of the topic is one of the main drivers of opinion formation. therefore, we manually compiled a set of contemporary discussion topics together with subreddits devoted to them ( for the study at hand, we selected the 8 most active topics, and for each of those, we extracted all users with at least 10 posts. the final dataset statistics are provided in ## stance polarity and intensity labels as the opinion of the users towards the investigated topics is a central dimension when studying the echo chamber phenomenon, we discarded our automated stance classification efforts (f1-score around 60% on three classes) and utilized the human labels of the spinos dataset instead @xcite 1 . since the spinos dataset resembles a proper subset of the data that will be studied in this paper, it was deemed a feasible source for human labeled user stance samples. the stance labels and their corresponding numeric values are: strongly against (-2), moderately against (-1), stance not inferrable (0), moderately in favor (1) and strongly in favor (2). the dataset consists of 3526 manually annotated posts from 640 users, which fully overlap with our reddit data. we analyzed the annotated stances of each user and verified that most users consistently persist on a particular stance polarity. there are a few users with vacillating stances, who seem to mostly persist on one pole (either in favor or against) and express strong stance intensity only for that pole. we therefore compute each user's average stance based on the individual stances of their posts. figure 1 shows the distribution of the averaged user stances for each topic. we note that, based on the annotation guidelines, a positive stance in the topic of feminism means being in favor of equal rights for all genders, a positive stance against climate change means believing that climate change is caused by humans and constitutes a potential threat on survival and a positive stance in the case of gun control means argumenting in favor of the public availability of guns. ## identifying echo chambers apart from the stance, a central aspect in the analysis of network structures, and especially echo chambers, in social media datasets is the definition of the interaction itself. researchers have used retweets @xcite or follows @xcite @xcite to represent edges between user nodes. these however do not involve an explicit effort of content production @xcite , which is why we, following the work of @xcite , focus on replies in the downward subtree of the post to extract the social network topologies. figure 2 shows an example of how the post-reply tree of a social media post is transformed into the connecting edges of a user interaction network. detection approach similar to the studies of @xcite or @xcite , we search for interconnected user groups to detect echo-chamber-resembling structures in the extracted interaction topology. we experimented with the louvain algorithm @xcite , the label propagation algorithm from @xcite , and the fluid algorithm @xcite . the latter yielded the best qualitative results. to choose the optimal amount of communities, we encompass the fluid community detection with a meta-algorithm based on the notion of modularity of the detected communities. the meta algorithm runs the fluid community detection on the extracted network graphs from 2 up until 7 communities, and keeps track of the modularity of the created partitions, which is defined as: in the end, the partition with the highest modularity score is returned. we chose modularity since it measures the division of the network into communities, i.e., whether there are only a few connections between the communities, while the nodes within them are densely connected @xcite , which aligns with our goal of finding echo chambers. to ensure the consistency of our results even despite the elements of randomness in the community detection, 30 runs of the detection function are performed for each community amount, while still maximizing for modularity. based on these created partitions, we capture three graph community metrics @xcite , in order to measure the degree to which each distinct community represents an echo chamber in the network topology on variables that are not explicitly optimized during the community detection. we utilize separability, expansion and density, defined as follows for a community c: the higher the values for separability and density of a detected community are, the more the interactions of its users are segregated from the rest of the network and rather take place with people from the same "bubble" and therefore represent an echo chamber. expansion resembles echo chamber effects anti-proportionally as it is increased, when users of a community have more interactions with members from the other groups. to visualize the network nodes, we use the fruchterman-reingold force-directed placement @xcite . we further provide the average manually annotated user stance, as described in section 4, for each of the detected interaction communities to explore if these also represent distinct stance clusters. additionally, a weighted average stance for a community is determined by weighting the sampled stance values by the node degree of the users that contribute them. ## socio-demographic prediction to study how these community structures relate to their participants' socio-demographic traits we train interpretable supervised classifiers on datasets from previous social-media (twitter) studies on gender, age, and political ideology @xcite @xcite . for all three of these dimensions, it has been previously shown that social media users cluster along their labels @xcite @xcite . following previous studies, and considering the available data volumes, we approach these tasks as classification. 2 we acknowledge the suboptimality of predicting binary gender labels and using self-reported training data with users having label / data tw1 we predict the user's: (i) self-reported age (three classes: below 30, between 30-45, and 46+), (ii) self-reported gender (male/female), and (iii) political ideology (conservative, moderate, or liberal). our training data from twitter for age and gender consists of 3960 users. in order to directly include also users from reddit in the training data, we employ an automatic annotation generation for the dimensions of gender and age group based on regex-matching of 'i am'-statements in the user posts @xcite . for instance we annotate gender by searching for statements such as 'i am a guy/girl' or age with phrases such as 'i am x years old' or 'my grandson/granddaughter'. users with multiple contradicting 'i am'-statements are excluded from the dataset. this way, we enhance our training data with 966 users from reddit, annotated for gender and 289 users for age. we then enhance the ideology training data with 1223 users from subreddits r/liberal and r/conservative, excluding users with less than 5 posts. feature settings (1) tf-idf: we use the porter stemmer together with the tf-idf weighting scheme @xcite . we intentionally apply only easily interpretable classification models; linear svm, logistic regression, and random forest. the best-performing setup for each of the user traits, which is used further in paper, is highlighted in ## discussion and limitations the topic-and platform-specific environment underlines the limits of text-based user studies such as ours, indicating a lexical issue in the predictors used, confirming our hypothesis (iv) that the automated socio-demographic profiling tools suffer from a strong topical bias. while words such as problem, understand, or politics tend to be in general statistically more often used by self-identified men ( ## summary and conclusions we explore the social media phenomenon of echo chambers with regards to its socio-demographic implications. to quantify the forming of these structures, we employed an interaction graph-based algorithm, exploring the separability, density and expansion of the detected communities. for the network topologies of abortion, capitalism, and feminism, we found a moderate to high resemblance of the echo-chamber phenomenon. bridging the gap between theory and practice, these algorithm and measures could also be used by actual socialmedia platforms to track where its communities are structurally 'echo-chambered' and potential counter-measures are needed. to capture the socio-demographic distributions of the detected communities, we trained interpretable socio-demographic estimation models, scrutinized by keyphrase-based approaches. by merging the network and content information, we found that more 'echo-chambered' topic communities also show an increased separation in their stance and gender and ideology profiles. these results reinforce the call for incorporating socio-demographic and network information into data sets and models for tasks like sentiment analysis, text generation and stance prediction @xcite , while keeping in mind that a lexical topicrelated bias can be a source of misinterpretation in domain-specific user modeling.