Enabling Cross-Platform Comparison of Online Communities Using Content and Opinion Similarity (2024)

Abstract: With the continuous growth of online communities, understanding their similarities and dissimilarities is more crucial than ever for enhancing digital interactions, maintaining healthy interactions, and improving content recommendation and moderation systems. In this work, we present two novel techniques: BOTS for finding similarity between online communities based on their opinion, and Emb-PSR for finding similarity in the content they post. To facilitate finding the similarity based on opinion, we model the opinions on online communities using upvotes and downvotes as an indicator for community approval. Our results demonstrate that BOTS and Emb-PSR outperform existing techniques at their individual tasks while also being flexible enough to allow for cross-platform comparison of online communities. We demonstrate this novel cross-platform capability by comparing GAB with various subreddits.

## introduction online communities have become important venues for public discourse and opinion. platforms like reddit, with its diverse range of subreddits, offer a unique window into the collective sentiments and interests of different groups. understanding distribution statement a. approved for public release. distribution is unlimited. this material is based upon work supported by the department of the air force under air force contract no. fa8702-15-d-0001. any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the department of the air force. delivered to the u.s. government with unlimited rights, as defined in dfars part 252.227-7013 or 7014 @xcite . notwithstanding any copyright notice, u.s. government rights in this work are defined by dfars 252.227-7013 or dfars 252.227-7014 as detailed above. use of this work other than as specifically authorized by the u.s. government may violate any copyrights that exist in this work. while several prior works have explored the task of measuring the similarity between communities @xcite @xcite , existing works either measure the community similarity in only a single dimension or use manually annotated seed communities to guide similarity measures across different dimensions. in contrast, our approach divides the community similarity task into two distinct components -content similarity and opinion similarityand does not require manually identified seed communities. content similarity measures the similarity in the types of topics being discussed in the two communities regardless of how a community reacts towards a topic. for example, all politics related subreddits will be close in this metric regardless of their political leaning. in contrast, opinion similarity measures the similarity in the way communities react to content posted on the community. for example, left-leaning politics related subreddits will be close to each other but will be further away from right-leaning subreddits. we obtain content similarity between communities a and b by calculating the ratio of posts in a community b that are similar to posts in a. we use sentencebert to obtain sentence embeddings to facilitate the comparison. to enable the calculation of opinion similarity, we model the opinions of a community by training a llama2 @xcite model at the task of predicting whether a comment will get heavily upvoted or downvoted. if a model trained on data from subreddit a can accurately predict the upvote/downvote patterns in subreddit b, we infer that the two subreddits share similar opinions. we evaluate content similarity and opinion similarity separately and demonstrate that our techniques outperform existing techniques by a significant margin in both the tasks. our meth-ods, while being more performant also have the unique advantage of allowing us to compare online communities from different social media platforms. we demonstrate this benefit by comparing discussions from the gab online community with various subreddits. more concretely, our contributions are as follows. ## related work numerous prior studies have explored the task of forecasting content popularity on social media platforms. the social media prediction (smp) challenge @xcite predicts image popularity on flickr, leveraging both text and visuals. @xcite predict the popularity of posts by extracting features from textual and visual components, incorporating numerical attributes and follower and following counts, and leveraging a catboost model. other related works make use of neural network architectures @xcite @xcite . notably, visual features are commonly derived through convolutional neural networks, while text features frequently employ pretrained embeddings, such as word2vec @xcite . several prior works have explored the task of representing a community. @xcite creates embedding representations for communities by applying the glove algorithm @xcite on a user-subreddit co-occurrence matrix. they apply t-sne (van der @xcite on the resulting embeddings to find clusters of similar subreddits. in a similar vein, @xcite create community embeddings by using word2vec on data where the communities are treated as words and users are treated as contexts. @xcite use the same word vectors and apply agglomerative clustering to group subreddits; they use manually identified seeds to order the communities along various "social dimensions" such as age, partisanship, and affluence. @xcite use the vectors developed by @xcite to rank the subreddits on a political scale based on their similarity to r/conservative and r/progressive subreddits. @xcite use an approach similar to @xcite to create embeddings for each subreddit. the distance between subreddits is measured by the cosine distance between the l2-norm of any two subreddit embeddings. @xcite use a graph neural network to model communities and perform tasks such as violation detection, sentiment and analysis. in our work, we model the opinions of a community by training a llama2 model to predict whether a comment will get heavily upvoted or downvoted. @xcite cluster subreddits based on whether a model trained on moderation data for each make similar predictions. @xcite model reddit conflicts by clustering communities by applying word2vec or glove to co-occurrences of actively posting users among subreddits. @xcite identify intercommunity conflicts by identifying users who receive many upvotes in one subreddit and receive many downvotes in another subreddit. teblunthuis and hill (2022) cluster subreddits based on user overlap, which is high if two subreddits share a lot of users. @xcite assign subreddits along a "conspiracy scale" by applying pca on the user-subreddit co-occurrence matrix and manually identifying the axis that ranks known conspiracy and science subreddits on opposite ends. @xcite , while studying the rise of conspiracy theories in the aftermath of major events, use positive pointwise mutual information and com-munity2vec @xcite vectors to identify communities similar to r/science and r/conspiracy. in contrast to past work, our approach to measuring community similarity does not require manually identified seed communities. once an opinion model is trained on a subreddit, such as r/science, we can subsequently determine its similarity to other online communities without any further retraining. ## corpus overview we use data from two popular online social media platforms, reddit and gab, in our experiments. we obtained data from both platforms through pushshift @xcite , including all posts and comments made on reddit each month. our models are trained on reddit data between january 2018 and april 2023. our focus is on assessing the approval or disapproval comments receive within specific subreddits, given the context of individual posts. for each post, we use its title, id, and selftext (i.e., textual content). for each comment, we extract the comment's text and its score, calculated as the number of upvotes minus the number of downvotes. upvotes and downvotes signify community approval or disapproval, respectively, with each account limited to one vote per post and comment. while multiple accounts are permissible @xcite , using them for vote manipulation can lead to temporary or permanent banning of a user's reddit account @xcite . posts and comments with a high score gain more visibility within a community. in contrast with prior work, we intentionally exclude network-based features (e.g., author name and interaction history) to ensure our methods' applicability across platforms like reddit and gab. pushshift's gab data, whose post objects more closely resemble twitter tweet objects (twitter api, 2024), lacks the post title and score attributes seen in reddit data collected through pushshift. to account for this change, we reuse the text body of a gab post object for both the title and selftext fields when passing gab data through model trained on reddit. ## predicting opinion polarity of individual comments one of our goals is to quantify the similarity of subreddits based on their opinions and not merely the textual content. as a first step towards this goal, in this section we devise a method to model the opinion of distinct communities. for a given comment on a specific post, we treat its score as a self-labelled ground truth data set for that community's opinion. class l posts indicate diapproval, class h posts indicate approval; see section 3.1. we build the comment classification model using llama2. our model consists of the llm followed by a single linear layer and a softmax layer for classification. the last token's vector representation is passed to the linear layer as the sentence embedding. the input to the model consists of the post title, selftext (post text), and the text of the comment being classified. these inputs are passed into the llm in the following format: finally, the model predicts either low or high. we trained llama2 models until convergence or up to 20 epochs. due to limitations of our computational resources, we experimented with only the 7 billion parameter version of llama2. fine-tuning was accomplished using 16-bit parameters and the lora technique @xcite , a parameter-efficient training strategy. we compare the lora finetuned model against llama2-7b under 0-shot and 5-shot setting, below. in the 5shot setting, five random {post title, selftext, com-ment} pairs from the same subreddit were used. the specific prompt format for the 5-shot setting is detailed in appendix a. the code used to train and evaluate our models can be found in our github repository 1 . ## community similarity based on both content and opinion as discussed in section 2, the results from the existing community similarity metrics focus heavily on content-based similarity, or they rely on manually selected set of seed communities to calculate opinion similarity. furthermore, none of the existing community similarity metrics allow for cross platform (e.g. reddit, gab, or twitter) comparison of communities. in reality, subreddits can differ by opinion even if they are similar in terms of a topic. in this section, we propose to quantify similarity among subreddits (or any collection of posts) according to both content and opinion. figure 1 shows the results of our approach, for a methodology we detail below. subreddits that are similar to the gab in terms of posted content appear closer to 1 on the x-axis, explained in section 5.1; whereas subreddits that are similar to opinions expressed on gab appear closer to 1 on the y-axis, explained in section 5.3. ## cross platform community similarity we intentionally omit features derived from social network interactions, such as usernames and usercommunity engagement patterns, from our community similarity calculations. this enables us to compare communities beyond the constraints of a single social network. to our knowledge, existing community similarity methods do not readily facilitate cross-platform comparisons. we demonstrate the cross-platform capabilities of both bots and emb-psr by comparing gab with various subreddits. to compare gab with various subreddits, we have two options: train an opinion classification model on gab data and then predict opinions on subreddit data, or load pre-trained subreddit models and use them to predict opinions on gab data. we opted for the latter approach to avoid the computationally intensive task of training a new model. as detailed in section 3, our gab dataset, obtained via pushshift, spans from june 2018 to october 2018. applying similar preprocessing steps as with the reddit data, we randomly sampled 10,000 posts with a score of at most -2 and 10,000 posts with a score of at least 25. we then loaded pre-trained subreddit models and had them predict the score classes of these 20,000 posts to obtain the similarity (f 1 score on score classification) between gab and other subreddits. however, post structure differs between reddit and gab. reddit includes comments within the context of a post and selftext, while gab posts in our dataset are independent and lack selftext. to address this, we found that repeating the gab post text as both the selftext and comment was sufficient for comparison with subreddits. we present the results in a "similarity plot" in figure 1 . the x-axis represents content similarity (emb-psr) and the y-axis represents opinion similarity (bots). we observe that gab is similar in content to several politics related subreddits including r/conservatives, r/neoliberal, and r/republican. among these subreddits, however, its opinions appear more similar to right-leaning subreddits (e.g. r/conservatives, r/askthe_donald) compared to left-leaning subreddits (e.g. r/neoliberal, r/socialism). the r/theleftcantmeme, a right-leaning subreddit, and r/therightcantmeme, a left-leaning subreddit, demonstrate the usefulness of a similarity plot -the two subreddits are very close in terms of their content similarity to gab, but gab is much more similar to r/theleftcantmeme in its opinions compared to r/therightcantmeme. bots and emb-psr, when presented in a similarity plot, allow us to analyze subreddit similarity in a more holistic manner by considering both content and opinion similarity. ## conclusion our research advances the understanding of online communities and their similarities. by training llama2 models on the task of comment score classification, we demonstrate their impressive performance across diverse subreddits, showcasing their potential to accurately predict community reactions. we decompose the task of calculating community similarity into two fundamental components: opinion similarity and content similarity. we then introduce two novel techniques to quantify these dimensions within online communities. our evaluation demonstrates that our techniques outperform existing community similarity measures by a considerable margin on their individual tasks. furthermore, our similarity measures are less dependent on platform-specific features and do not require manually identified seed communities, enabling cross-platform comparison of communities as demonstrated through our comparison of gab with various subreddits. this research paves the way for a more comprehensive understanding of online communities, with potential applications in a broad range of tasks ranging from content moderation to community recommendation.