B ot P ercent: Estimating Bot Populations in T witter Communities (2023)

Abstract: Twitter bot detection is vital in combating misinformation and safeguarding the integrity of social media discourse. While malicious bots are becoming more and more sophisticated and personalized, standard bot detection approaches are still agnostic to social environments (henceforth, communities) the bots operate at. In this work, we introduce community-specific bot detection, estimating the percentage of bots given the context of a community. Our method—BotPercent—is an amalgamation of Twitter bot detection datasets and feature-, text-, and graph-based models, adjusted to a particular community on Twitter. We introduce an approach that performs confidence calibration across bot detection models, which addresses generalization issues in existing community-agnostic models targeting individual bots and leads to more accurate community-level bot estimations. Experiments demonstrate that BotPercent achieves state-of-the-art performance in community-level Twitter bot detection across both balanced and imbalanced class distribution settings, presenting a less biased estimator of Twitter bot populations within the communities we analyze. We then analyze bot rates in several Twitter groups, including users who engage with partisan news media, political communities in different countries, and more. Our results reveal that the presence of Twitter bots is not homogeneous, but exhibiting a spatial-temporal distribution with considerable heterogeneity that should be taken into account for content moderation and social media policy making. The implementation of BotPercent is available at https://github.com/TamSiuhin/BotPercent.

## introduction twitter accounts controlled by automated programs, also known as twitter bots, have become a widely recognized, concerning, and studied phenomenon @xcite . * these authors contributed equally to this work. ## botpercent methodology we propose botpercent, a novel system for community-level twitter bot detection. as illustrated in figure 2 , botpercent first trains multiple models on a mixture of datasets, then leverages confidence calibration to perform unbiased bot percentage estimation, and finally combines predictions with learnable weights. ## analysis of twitter bots section 3 demonstrates that botpercent achieves state-of-the-art performance on community-level bot detection and produces well-calibrated bot percentage estimation for diverse twitter communities. we now investigate diverse real-world twitter communities and estimate their bot populations by leveraging botpercent. ## bot population in different countries' politics existing research on the twitter bot population mostly focuses on bots in u.s. politics @xcite while neglecting the political landscape of other countries that could have similar problems. we complement the scarce literature by investigating the bot population in different countries' political communities. specifically, we use the president or prime minister's twitter account as a starting point and sample their followers to serve as a proxy for the politically engaged communities in different countries. figure 9 7 illustrates that the percentage of bots in u.s. politics is the highest, while other english-speaking countries also witness higher levels of bot interference. in addition, the percentages of bots in the political communities of argentina, france, and nigeria are the lowest, indicating a more genuine and authentic political discourse. these results again reaffirm that twitter bots have spatial pat- ## ablation study as botpercent outperforms various baselines in community-level bot detection tasks, we investigate the impact of each module in botpercent to verify their effectiveness. more specifically, we perform ablation studies on feature-, text-, and graph-based modules and test the effectiveness of calibration and weighted sum combination, as is shown in ## related work existing twitter bot detection models mainly fall into feature-based, text-based, and graph-based. feature-based bot detection methods extract features from user timelines and metadata while using traditional classification algorithms to identify bots. features can be extracted from various sources such as metadata @xcite , user description @xcite , tweets @xcite , temporal tweeting patterns @xcite , and the following relationship @xcite . later research focused on improving the scalability of featurebased approaches @xcite , automatically discovering new bots @xcite , and finding the optimal balance between precision and recall @xcite . however, as bot operators become more aware of the features used by feature-based methods, they try to alter them to evade detection @xcite , which makes it difficult for feature-based methods to effectively identify advanced bots. text-based methods propose to leverage techniques in natural language processing to analyze tweets and user descriptions. these methods employ word embeddings @xcite , recurrent neural networks @xcite , the attention mechanism @xcite , and pretrained language models @xcite to analyze twitter text. later research efforts combine tweet representations with user features @xcite , learn unsupervised user representations @xcite , and address the issue of multilingual tweets @xcite . however, advanced bots counter text-based approaches by diluting malicious tweets with content stolen from genuine users @xcite , while @xcite found that relying solely on tweet content may not be robust or accurate enough to win the bot detection arms race. graph-based methods for twitter bot detection interpret twitter as a network and leverage concepts from network science and geometric deep learning. these approaches adopt node centrality @xcite , representation learning @xcite , graph neural networks (gnns) @xcite @xcite , and heterogeneous gnns @xcite @xcite . researchers have also explored combining graph-and text-based methods @xcite or proposing new gnn architectures to take into account heterogeneities in the twitter network @xcite . these graph-based approaches have been effective in addressing the challenges such as bot clusters and bot disguises @xcite . there are also preliminary attempts to ensemble methods from different categories in bot detection. for example, ng and carley (2022) trains featureand text-based experts for ensemble, and @xcite further ensembles three models each from feature-, text-, and graph-based categories to boost community-agnostic bot detection. however, to improve stability and generalizability, the ensemble size for both data and model needs to be increased. although these community-agnostic approaches have advanced the understanding of twitter bots and their behavior, community-level bot detection, aiming to understand the extent to which bots compromised a given twitter community, remains an underexplored yet important problem. with great implications for both platform moderators and everyday users, we propose a novel community-level system botpercent with multidataset training and the largest model ensemble to estimate the twitter bot populations from groups to crowds. ## conclusion we propose botpercent, a multi-dataset multimodel twitter bot detection system for estimating the twitter bot populations from groups to crowds. experiments on the twibot-22 benchmark demonstrate that botpercent achieves state-of-theart performance on community-level bot detection while being significantly more robust to perturbations in user features. armed with botpercent, we investigate the bot populations in twitter communities, such as news commentators, politically engaged accounts, and more. together these results demonstrate that the existence of twitter bots is not homogeneous, rather having spatial-temporal patterns that have great implications for social media moderation, warning of socio-political events, real-time monitoring, and botpercent's potential to guide future research in specific communities rather than general-purpose.