Towards Detecting Contextual Real-Time Toxicity for In-Game Chat (2023)

Abstract: Real-time toxicity detection in online environments poses a significant challenge, due to the increasing prevalence of social media and gaming platforms. We introduce ToxBuster, a simple and scalable model that reliably detects toxic content in real-time for a line of chat by including chat history and metadata. ToxBuster consistently outperforms conventional toxicity models across popular multiplayer games, including Rainbow Six Siege, For Honor, and DOTA 2. We conduct an ablation study to assess the importance of each model component and explore ToxBuster’s transferability across the datasets. Furthermore, we showcase ToxBuster’s efficacy in post-game moderation, successfully flagging 82.1% of chat-reported players at a precision level of 90.0%. Additionally, we show how an additional 6% of unreported toxic players can be proactively moderated.

## introduction online spaces are plagued by toxic speech, spanning social media platforms (e.g., facebook @xcite , twitter @xcite , reddit @xcite , youtube @xcite ), in-game chats @xcite and news websites comment sections @xcite . as evidenced by surveys from the anti-defamation league (adl), exposure to toxic language not only alienates users but also poses a range of psychological harms and the potential to incite real-world violence @xcite . even worse, marginalized groups continue to face a disproportionate level of targeted online hate and harassment. companies strive to foster a healthy online community and have employed various methods to address toxic speech, such as censoring words, (shadow) banning users or blocking them from communicating @xcite . however, the vast amount of user-generated data and the rapidly evolving nature of language have made it exceedingly challenging to implement consistent moderation practices. we leverage recent advancements in large language models to create accurate and transferable models for effective content moderation. contextual language embeddings, like bert @xcite , serve as the foundation for many state-ofthe-art toxic speech detection models. however, most existing approaches either neglect context entirely or yield only marginal improvements and are not fit for real-time moderation for in-game chat. to address this limitation, we propose tox-buster, the first real-time in-game chat toxicity detection model capable of integrating chat history and metadata. it is trained on annotated datasets that encompass diverse perspectives, with annotators from marginalized groups offering valuable insights. furthermore, we demonstrate the transferabilty of toxbuster across different games, as well as its adaptability to a completely different domain, i.e., comment threads of news articles. when it comes to post-game moderation, we prioritize high precision settings to maximize the impact of our model without overloading limited manual content moderation resources with false positives. remarkably, toxbuster achieves an impressive 82% identification rate for toxic players-reported by other players for posting toxic chat-at a precision level of 90.0%. this can significantly reduce the moderation load of human moderators. in summary, contributions are threefold: reproducibilty: source code for toxbuster is hosted on github. due to legal and privacy reasons, r6s and fh datasets cannot be public. ## related works toxicity detection research has gained increasing attention due to the challenges it poses. these challenges include not only the text itself, such as the presence of out-of-vocabulary words, but also the lack of consensus on the precise definition of toxicity @xcite . definitions vary, ranging from a broad concept of toxicity @xcite to more specific categories like hate speech @xcite , abusive language (park and fung, 2017), cyberbullying @xcite , and offensive language. in our study, we focus on adapting categories defined by alliance (2020) to address disruptive behavior in online games. traditionally, toxicity detection has been approached as a classification task using various methods, including traditional ml models with manual feature engineering @xcite , deep neural networks @xcite , and pretrained language models @xcite @xcite . previous studies have explored including additional context such as news article titles @xcite , usernames @xcite , twitter user metadata (fehn unsvåg and gambäck, 2018), game-specific slang @xcite and parent comments and discussion titles @xcite . these attempts to incorporate additional context into language models have yielded limited performance gains, typically less than 1%. to address this, we incorporate more than just the previous comment @xcite by utilizing all preceding chat history available as well. while previous research on toxicity detection has explored metadata aspects, none have addressed the conversational aspect, especially with this level of granularity, in contexts where more than just the preceding line of text is used. we draw inspiration from multi-turn conversational models and incorporate speaker segmentation (metadata including teamid, chat type, and playerid) and a naive dialogue augmentation technique that has shown to enhance bert for multi-turn conversations @xcite . ## methodology in this section, we present four toxicity datasets along with toxbuster and the baselines models. ## results and discussions in the upcoming sections, we analyze toxbuster's performance relative to baselines, examine individual toxic class performance, conduct an ablation study to evaluate each model component's impact and assess implications for post-game moderation. ## conclusion in this paper, we propose toxbuster, a simple and scalable model specifically designed for real-time chat moderation in in-game chat. toxbuster outperforms baseline models, including cleanspeak and perspective api, across multiple datasets and toxic categories. additionally, toxbuster can be easily adapted for comment threads of social media platforms. our study emphasizes the importance of using up-to-date toxic game chat datasets to enhance model robustness, rather than relying on generic toxicity datasets. furthermore, we demonstrate the seamless transferability between r6s and fh datasets. ablation studies further validate the significant contributions of chat history and speaker segmentation to toxbuster's effectiveness. the precision-recall analysis illustrates tox-buster's capability to accurately identify and flag toxic content with high precision and recall. in an automated post-game moderation scenario, tox-buster successfully identifies toxic chat players, flagging over 41% of chat-reported players at a precision level of 99.9%. additionally, we explore how toxbuster can partially address the issue of toxic players who are not chat-reported. overall, toxbuster presents a robust and efficient solution for automating chat moderation, promoting a safer and more positive gaming experience for players.