When the Echo Chamber Shatters: Examining the Use of Community-Specific Language Post-Subreddit Ban (2021)

Abstract: Community-level bans are a common tool against groups that enable online harassment and harmful speech. Unfortunately, the efficacy of community bans has only been partially studied and with mixed results. Here, we provide a flexible unsupervised methodology to identify in-group language and track user activity on Reddit both before and after the ban of a community (subreddit). We use a simple word frequency divergence to identify uncommon words overrepresented in a given community, not as a proxy for harmful speech but as a linguistic signature of the community. We apply our method to 15 banned subreddits, and find that community response is heterogeneous between subreddits and between users of a subreddit. Top users were more likely to become less active overall, while random users often reduced use of in-group language without decreasing activity. Finally, we find some evidence that the effectiveness of bans aligns with the content of a community. Users of dark humor communities were largely unaffected by bans while users of communities organized around white supremacy and fascism were the most affected. Altogether, our results show that bans do not affect all groups or users equally, and pave the way to understanding the effect of bans across communities.

## introduction online spaces often contain toxic behaviors such as abuse or harmful speech @xcite @xcite @xcite @xcite @xcite @xcite . such toxicity may result in platform-wide decreases in user participation and engagement which, combined with external pressure (e.g., bad press), may motivate platform managers to moderate harmful behavior @xcite . moreover, the radicalization of individuals through their engagement with toxic online spaces may have real-world consequences, making toxic online communities a cause for broader concern @xcite @xcite . reddit is a social media platform that consists of an ecosystem of different online spaces. as of january 2020, reddit had over 52 million daily active users organized in over 100,000 communities, known as "subreddits", where people gather to discuss common interests or share subject-or format-specific creative content and news @xcite . every post made on reddit is placed in one distinct subreddit, and every comment on reddit is associated with an individual post and therefore also associated with a single subreddit. as reddit continues to gain popularity, moderation of content is becoming increasingly necessary. content may be moderated in several ways, including: (1) by community voting that results in increased or decreased visibility of specific posts, (2) by subredditspecific volunteer moderators who may delete posts or ban users that violate the subreddit guidelines, and (3) by platform-wide administrators that may remove posts, users, or entire communities which violate broader site policies. the removal of an entire subreddit is known as a "subreddit ban," and does not typically indicate that the users active in the subreddit have been banned. given that the ostensible purpose of subreddit bans is to remove subreddits that are in habitual noncompliance with reddit's terms of service, it is important to understand whether such bans are successful in reducing the offending content. this is especially of interest when the offending content is related to harmful language. though limited, there is some evidence to suggest that subreddit bans may be effective by certain metrics. past work has demonstrated that these bans can have both user-and community-level effects @xcite @xcite @xcite . several of these studies have suggested that (1) subreddit bans may lead a significant number of users to completely stop using the site, and that (2) following a ban, users that remain on the platform appear to decrease their levels of harmful speech on reddit @xcite @xcite . @xcite also illustrated that postban migrations of users to different subreddits did not result in naive users adopting offensive language related to the banned communities. more work is required to better understand changes in the language of individual users after such bans. ## previous work previous research provides a foundation for investigating the effects of subreddit bans on harmful language and user activity. detection of offensive content typically takes the form of automated classification. different machine learning approaches have been applied with varied success, including but not limited to support vector machines and random forests to convolutional and recurrent neural networks @xcite @xcite @xcite @xcite @xcite . more recently, @xcite used an ensemble learning algorithm to classify both hate speech and counter speech in a curated collection of german messages on twitter. unfortunately, these approaches require labeled sets of speech to train classifiers and therefore risk not transferring from one type of harmful speech (e.g. misogyny) to another (e.g. racism). we therefore aim for a more flexible approach that does not attempt to classify speech directly, but rather identifies language over-represented in harmful groups; i.e., their in-group language. that language is not a signal of, for example, hate speech per se. in fact, any group is likely to have significant in-group language (e.g. hockey communities are more likely to use the word "slapshot"). however, detection of in-group language can be fully automated in an unsupervised fashion and is tractable. the majority of past work on bans of harmful communities on reddit only examined one or two subreddits, often chosen due to notoriety @xcite @xcite @xcite . many of these studies focused on the average change in behavior across users and did not consider the factors which may drive inter-individual differences in behavior following a ban @xcite @xcite . different users may respond differently to subreddit bans based on their level of overall activity or community engagement. for example, @xcite found that users that were more active on reddit prior to a subreddit ban were more likely to migrate to a different platform following a ban. a user's activity levels prior to a ban also impacted whether activity levels increased or decreased upon migrating to a different platform @xcite . @xcite demonstrated that users who were more active in a subreddit prior to a ban were more likely to change their behavior following the banning of that subreddit, but the authors did not investigate the ways in which users changed their behavior. lastly, hazel kwon and shao (2020) found that a user's pre-ban activity level within r/alphabaymarket in-fluenced post-ban shifts in communicative activity. while we are interested in the effects of moderation on any online community, we study reddit because the platform is strongly partitioned into sub-communities, and historical data on both subreddits and users are readily available @xcite . reddit users are regularly active in multiple subreddits concurrently, and unlike other sub-community partitioned platforms like discord, slack, or telegram, we can easily retrieve a user's activity on all sub-communities. this provides an opportunity to understand how the members of a community change their behavior after that community is banned. furthermore, knowledge of the drivers of inter-individual behavioral differences may permit moderators to monitor the post-ban activity of certain subsets of users more closely than others, which may lead to an increase in the efficacy of platform-wide moderation. ## methodology as part of investigating whether different communities respond differently to a subreddit ban, we examine whether top users differ from random users in their change in activity and in-group language usage following community-level interventions. specifically, we utilize natural language processing to track community activity after a subreddit ban, across 15 subreddits that were banned during the so-called "great ban" of 2020. we first identified words that had a higher prevalence in these subreddits than on reddit as a whole prior to a ban. these words do not necessarily correspond to harmful speech but provide a linguistic signature of the community. the strengths and drawbacks of this approach are discussed in the discussion and appendix. we then compared the frequency of use of community-specific language, as well as the overall activity level of a user (i.e., the number of total comments), 60 days pre-and post-ban for (1) the 100 users that were most active in the banned subreddit 6 months prior to the ban and (2) 1000 randomly sampled non-top users. we predicted that top and random users that remained on the site following a subreddit ban would react differently to the ban, and we anticipated that there would be variation in how different communities responded to a ban. ## results by comparing the median change in activity and vocabulary usage among top and random users, we found a consistent pattern: top users, for every subreddit studied, decrease their activity more than their peers. this result is important to keep in mind when a uniform sampling of subreddit users post-ban may indicate that a community ban was ineffective. we do not find as consistent a difference between top and random user when looking at vocabulary change; suggesting that while bans may drive harmful users to inactivity, they are less clearly effectual at reforming user behavior. these results are summarized in fig. 2a . to confirm our findings, we tested the statistical significance of differences between top and random distributions for each subreddit, illustrated in fig. 3 . in all subreddits, there was a significant difference between top and random user changes in either activity shifts, vocabulary shifts, or both. considering a significance threshold on the false discovery rate, @xmath0.05, we found two subreddits (r/ccj2 and r/hatecrimehoaxes) that show significant differences in both shifts. the subreddit r/darkjokecentral shows significant differences between top and random users in vocabulary shift, but not activity; whereas the rest of the subreddits show differences in activity but not vocabulary shift between top and random users. we found that, controlling for user engagement (i.e., whether a user was a top or random user), there was a significant overall effect of subreddit identity on both the proportion of a user's total posts that occurred postban vs preban (lrt, @xmath1.730, @xmath2.001) and the proportion of a user's total in-group vocabulary that occurred postban vs preban (lrt, chi-@xmath3.680, @xmath4.001). controlling for subreddit identity, there was also a significant overall effect of user engagement on the proportion of a user's total posts that occurred postban vs preban (lrt, chi-@xmath5.452, @xmath6.001) and the proportion of a user's total in-group vocabulary that occurred postban (lrt, chi-@xmath7.020, @xmath8.001). postban posts made up a lower proportion of a user's total posts and postban use of in-group vocabulary made up a lower portion of a user's total in-group vocabulary use for top users compared to random users (fig. 4 ). there were a few subreddits that were significantly different from most or all of the other subreddits. for example, in r/the donald, postban posts comprised a higher proportion of a user's total posts, compared to all other subreddits (fig. 4a ), and postban use of in-group vocabulary comprised a higher portion of a user's total in-group vocabulary use, compared to all other subreddits (fig. 4b ). postban posts also comprised a higher proportion of a user's total posts in r/oandaexclusiveforum, com-pared to most other subreddits, while postban posts comprised a lower proportion of a user's total posts in r/soyboys, compared to most other subreddits (fig. 4a ). the proportion of a user's total in-group vocabulary that occurred postban was lower for both r/gendercritical and r/hatecrimehoaxes, compared to most other subreddits (fig. 4b ). ## discussion past work has been quick to conclude that subreddit bans either are @xcite @xcite or are not @xcite effective at changing user behavior. we have found that results differ between subreddits and between more and less active users within a subreddit. since many prior studies on banning efficacy focus on one to two subreddit case studies, these distinctions may not have been apparent in some previous datasets. to automatically study a larger number of communities, we tackle the simpler problem of tracking user activity and use of in-group language rather than more subjective harmful language. this approach has strengths and drawbacks. on the one hand, in-group language is easier to automatically identify with little expert knowledge or human intervention, while also including lesser known slang terms or dog whistles that could be harmful. on the other hand, our approach requires a large reference corpus that controls for relevant features of the category activity impact vocabulary impact dark jokes minimal minimal anti-political top users less active decrease among top users mainstream right wing minimal inconsistent extreme right wing all users decrease, especially top users minimal our study examines 15 subreddits with over 5000 daily users that were banned simultaneously after a change in reddit content policy, and our results suggest that subreddit bans impact top and random users differently (in agreement with prior studies such as hazel kwon and shao ( 2020) additionally, we see patterns in subreddit responses to bans that loosely correlate with the type of content the community focused on, summarized in ## conclusion we have provided a broad investigation of the impact of banning online communities on the activity and in-group vocabulary of the users therein. our work expands the scope of other studies on this subject, both in terms of the number and types of communities examined. through this more comprehensive analysis, we have demonstrated heterogeneity in the impact of bans, depending on the type of subreddit and the level of user engagement. we found that top users generally showed greater reductions in activity and in-group vocabulary usage, compared to random users. we also found that the efficacy of banning differs across subreddits, with subreddit content potentially underlying these differences. however, while we provide strong evidence of heterogeneity in ban efficacy, even more comprehensive research must be conducted on a larger group of subreddits in order to fully understand the dynamics behind this heterogeneity.