Multilingual Bot Accusations: How Different Linguistic Contexts Shape Perceptions of Social Bots (2024)

Abstract: Recent research indicates that the online use of the term ”bot” has evolved over time. In the past, people used the term to accuse others of displaying automated behavior. However, it has gradually transformed into a linguistic tool to dehumanize the conversation partner, particularly on polarizing topics. Although this trend has been observed in English-speaking contexts, it is still unclear whether it holds true in other socio-linguistic environments. In this work we extend existing work on bot accusations and explore the phenomenon in a multilingual setting. We identify three distinct accusation patterns that characterize the different languages.

## introduction social bots are described in academia as automated accounts in online media that have the ability to manipulate public opinion on large scale. while a plethora of current work on bots focusses on their detection, researchers are increasingly interested in how people perceive social bots and how they talk about them in the online sphere. in this work we extend a recent study by @xcite who look at accusation situations, i.e. those instances where a user on twitter (x) accuses another user of being a bot. while earlier work only focused on the english speaking landscape of the platform we want to investigate the phenomenon in a multi-lingual setting, examining the following research question: what are the differences in how users accuse each other of being a bot across different languages, and can we identify a taxonomy of these accusation types? 2 related work up until recently, the largest part of research on social bots has been concerned with developing meth-ods for their detection, trying to expose and characterize their efforts to systematically influence the discourse on social media @xcite . while most of these approaches focus on the platform twitter and only look on the english parts of it by default @xcite , few methods are available that are by design multilingual @xcite , helping to grapple with a phenomenon that has been found to impact the online public discourse and events in countries around the world @xcite , cascading across cultural and linguistic borders @xcite . while researchers developing these methods have to start out from the difficulty of operationalizing a phenomenon that has been found to be changing in meaning even within the academic discourse @xcite , recent work by @xcite has found an even more drastic shift in the understanding of the concept bot by social media users, away from the academic definition of a program that automatically produces content and interacts with humans on social media @xcite towards one that locates bots in the context of polarizing debates and insulting or even dehumanizing comments, effectively denying a (human) social media user their ability to meaningfully participate in the public discourse. their empirical finding, restricted to the english linguistic context, has been backed up by evidence from a survey conducted by @xcite , who report that more than a third of participants define bots as "fake accounts", "posing as actual humans", thereby trying to "sway public opinion". the findings by @xcite further indicate that users might be impacted by the threat they perceive from bot campaigns on social media by exposure to news coverage on them, opening up a channel for different perceptions across regional contexts. given such conceptual difficulties, it might not come as a surprise that researchers report for different platforms and linguistic and cultural contexts, including russian @xcite , chinese @xcite and english @xcite , how platform users struggle to distinguish human from bot accounts. on top of this general confusion, recent research on human perceptions of bots has found that users tend to perceive accounts with opposing viewpoints as non-human @xcite , easily dismissing their opinions @xcite . ## data to extend the study conducted by @xcite to a multi-lingual context, we combine different datasets collected and annotated for complimentary purposes. ## methods the core challenge in multilingual research is the handling of texts that the researcher is not familiar with. in our particular endeavor of studying the linguistic phenomenon of bot accusations across different languages, the ability to reliably discern tweets in which other users are actually accused of being bots from mere discussions of the concept or unrelated posts on potential synonyms is essential to the subsequent analysis methods, which, nonetheless, also need to be adjustable to a multilingual setting. ## results in the following, we first evaluate the performance of the different accusation detection methods, before presenting the analysis results on the accusations detected via our method of choice. ## discussion in this study, we expand on existing research about social bot accusations by examining linguistic settings beyond english. we developed an ensemble of language-specific and translation-based models to detect bot accusations in nine different languages. using this approach, we identified bot accusations on twitter (x) for each language from 2011 to 2023. our findings reveal that the previously noted shift in bot accusations in english 2012 2014 2016 2018 2020 2022 does not occur in every language. specifically, discussions in east asian languages, such as korean and japanese, show different patterns of bot accusations, with a stronger focus on automationrelated topics, particularly in the context of gaming. in contrast, accusations related to polarizing political debates were seldom observed in these languages. on the other hand, we identified languages such as russian, in which bot accusations were consistently associated with insults. our findings have several implications. from a moderation perspective, it is important to understand that these accusations should not be treated equally. while it is true that we need to acknowledge that "bot" is often systematically used as an insult, delegitimizing users' opinions and thus undermining constructive dialogue, the context and connotation of such accusations can vary significantly across different languages and cultures and, therefore, require different moderation strategies. we therefore highlight the risk of detection systems trained on english data only to fall short in generalizing to other context, emphasizing the need for diverse linguistic training to ensure accuracy and fairness. ## ethical considerations for our empirical study of the bot accusation phenomenon across a range of different languages, we are relying purely on publicly-accessible, usergenerated twitter posts. using this type of data carries the usual privacy risks known from social media studies, which we, however, try to counteract by anonymizing all data immediately after its collection. we are not interested in studying individual-level bot accusations, but rather focus on societal-level patterns. we do not try to identify any of the individuals included in our datasets, and explicitly point out that the bot accusations we observe on the platforms are oftentimes directed towards actual human beings instead of automated accounts, as indicated both by our findings and those of @xcite . we therefore discourage anyone from trying to infer the status and degree of automation of an account from a bot accusation found in our datasets or elsewhere. in recruiting annotators and collecting their annotations, we followed the ethical considerations and best practices put forth by the platform provider prolific, 4 including the guarantee that every annotator would receive an hourly pay equivalent (far) exceeding the required minimum pay as well as the informed consent and the possibility to withdraw from participation. to follow established practice in sharing research data collected from social media and still ensure full reproducibility and transparency of our results, we invite other researchers to contact us to mutually explore potentials for collaboration and the sharing of our collected research data. ## limitations most of this paper's limitations originate from its main conceptual and methodological challenge, the inherent multilinguality of the ambitious endeavor to study the same social media phenomenon across nine different languages. while we tried to handle this challenge as carefully as possible, we acknowledge a number of limitations that we were not able to overcome. first, and maybe most importantly, we rely on a number of pre-trained resources, particularly models. while the impressive zero-shot performances of advanced llms like gpt-3.5 has already been widely reported and used across a number of tasks and languages, the use of such general purpose, 'black-box' models should still be met with high attention and increased scrutiny. we tried to counterbalance our reliance on the llm's annotations in our different detection methods by ensuring that we validate the resulting classifier, by evaluating its performance on datasets that have been annotated by humans, without the mediating effects of any other methods or models. we decided to use the llm-based method only after we observed satisfying performance relative to human annotations on the relevant task of identifying bot accusations across different languages. regarding the use of the llm for purposes of the (exploratory) data processing and analysis, particularly its role in labelling the different context clusters, we did not validate the model through crowd annotations, but rather relied on our own judgments and experiments. we manually checked many of the automatically annotated clusters, represented through their (translated) significant tokens, and found that the clustering decisions of the llm were sufficiently reliable, particularly given the open-ended, exploratory nature of the task. a further limitation lies in the difficulty of reliably detecting bot accusations from tweets that are merely mentioning the keyword bot. this is already a challenging task for a single language, and even more so for nine different languages. as argued above, we tried to optimize for recall, i.e., the inclusion of as many accusations as possible, through a very broad initial data collection strategy, before then optimizing for precision, i.e., making sure that the tweets we identify as bot accusations actually are accusing other users of being a bot. we do so through our choice for the model ensemble method, the bot accusation method with the best f1-scores overall and -conceptuallythe best setup to only label those tweets as accusations that are identified as such by different types of classifiers. related to the limitation of being unable to achieve perfect accuracy in the bot accusation detection task, we acknowledge systematic biases that might occur because of linguistic particularities of and around the term bot. for instance, the german keyword we used, "bot", does not just refer to the concept we are interested in with this paper, but also translates to "offered". similarly, the arabic keyword ‫"ﺑﻮﺕ"‬ may also translate to "boot". we hope that, again, our choice of the ensemble model ameliorates such translation issues by combining a classifier that acts upon the original tweets with another classifier that works on a translated version of it. we hope that this study, as imperfect as it might be, still helps to advance our collective understanding of this interesting phenomenon beyond the much studied, english-only part of a social media platform like twitter.