Analyzing Norm Violations in Live-Stream Chat (2023)

Abstract: Toxic language, such as hate speech, can deter users from participating in online communities and enjoying popular platforms. Previous approaches to detecting toxic language and norm violations have been primarily concerned with conversations from online forums and social media, such as Reddit and Twitter. These approaches are less effective when applied to conversations on live-streaming platforms, such as Twitch and YouTube Live, as each comment is only visible for a limited time and lacks a thread structure that establishes its relationship with other comments. In this work, we share the first NLP study dedicated to detecting norm violations in conversations on live-streaming platforms. We define norm violation categories in live-stream chats and annotate 4,583 moderated comments from Twitch. We articulate several facets of live-stream data that differ from other forums, and demonstrate that existing models perform poorly in this setting. By conducting a user study, we identify the informational context humans use in live-stream moderation, and train models leveraging context to identify norm violations. Our results show that appropriate contextual information can boost moderation performance by 35%.

## introduction interactive live streaming services such as twitch foot_1 and youtube live foot_2 have emerged as one of the most popular and widely-used social platforms. unfortunately, streamers on these platforms struggle with an increasing volume of toxic comments and norm-violating behavior. 4 while there has been extensive research on mitigating similar problems for online conversations across various platforms such as twitter @xcite (2) relationships between chats are less clearly defined. such differences make chats in the synchronous domain more difficult to be moderated by existing approaches. @xcite @xcite , reddit @xcite @xcite , stackoverflow @xcite and github @xcite , efforts that extend them to live streaming platforms have been absent. in this paper, we study unique characteristics of comments in livestreaming services and develop new datasets and models for appropriately using contextual information to automatically moderate toxic content and norm violations. conversations in online communities studied in previous work are asynchronous: utterances are grouped into threads that structurally establish conversational context, allowing users to respond to prior utterances without time constraints. the lack of time constraints allows users to formulate longer and better thought-out responses and more easily reference prior context. on the other hand, conversations on live streaming platforms are synchronous, i.e. in real-time, as utterances are presented in temporal order without a thread-like structure. context is mostly established by consecutive utterances @xcite . the transient nature of live-stream utterances en-courages fast responses, and encourages producing multiple short comments that may be more prone to typos (70% of comments are made up @xmath0. figure 1 shows an illustration of the contrasting temporal and length patterns between the asynchronous and synchronous platforms. owing to these different characteristics, we find that previous approaches for detecting norm violations are ineffective for live-streaming platforms. to address this limitation, we present the first nlp study of detecting norm violations in live-stream chats. we first establish norms of interest by collecting 329 rules from twitch streamers' channels and define 15 different fine-grained norm categories through an iterative coding process. next, we collect 4,583 moderated chats and their corresponding context from twitch live streams and annotate them with these norm categories ( §2.1- §2.3). with our data, we explore the following research questions: (1) how are norm violations in live-stream chats, i.e. synchronous conversations, different from those in previous social media datasets, i.e. asynchronous conversations?; (2) are existing norm violation or toxicity detection models robust to the distributional shift between the asynchronous and synchronous platforms? ( §3.1, §3.3); and (3) which features (e.g., context and domain knowledge) are important for detecting norm violation in synchronous conversations? ( §3.2) from our explorations, we discover that (1) livestream chats have unique characteristics and norm violating behavior that diverges from those in previous toxicity and norm-violation literature; (2) existing models for moderation perform poorly on detecting norm violations in live-stream chats; and ## normvio-rt to investigate norm-violations in live-stream chat, we first collect norm violations in real-time conversations (normvio-rt), which contains 4,583 norm-violating comments on twitch that were moderated by channel moderators. 5 an overview of our 5 please contact the authors for the anonymized study data. twitch top 200 streamers annotators gather chat rules twitch api & irc monitor moderation events (ban, timeout, delete) stage 1 moderated message stage 2 moderated message + chat context stage 3 moderated message + chat / video context norm rule examples incivility be nice discrimination no racism … … define norms map rules multi-class annotation + knowledge statement data collection procedure is illustrated in figure 2 . we first select 200 top twitch streamers and collect moderated comments from their streamed sessions ( §2.1). to understand why these chats are moderated, we collect chat rules from these streamers and aggregate them to define coarse and fine-grained norm categories ( §2.2). we design a three-step annotation process to determine the impact of the chat history, video context, and external knowledge on labeling decisions ( §2.3). lastly, we present analysis of the collected data ( §2.4). ## toxicity detection in live-stream chat in this section, we first check whether norm violation and toxicity detection models are robust to the distributional shift from asynchronous conversations to synchronous conversations and vice versa, and identify how important the context or domain knowledge are for detecting toxicity and norm violation in synchronous conversations. ## related work toxicity detection most toxic language data consists of explicit hate speech consisting of hate lexicons @xcite @xcite , group identifiers @xcite , and hateful phrase (silva beyond binary toxicity detection treating toxicity detection as a binary task may not be enough to understand nuanced intents and people's reactions to toxic language use @xcite . to holistically analyze toxicity, recent works take a more fine-grained and multidimensional approach: (1) explainability explains why a particular chat is toxic with highlighted rationales @xcite , free-text annotations of implied stereotype @xcite @xcite , or pre-defined violation norms @xcite . these explanations can be used not only to improve the performance of the toxicity detection model, but also to train models that generate explanations; (2) target identification finds the targets of toxic speech, such as whether the target is an individual or a group, or the name of the group (e.g., race, religion, gender) @xcite ; (3) context sensitivity determines toxicity by leveraging context, such as previous tweets @xcite , comments @xcite or previous sentences and phrases within the comments @xcite . they show that context can alter labeling decisions by annotators, but that it does not largely impact model performance @xcite @xcite ; (4) implication understands veiled toxicity that are implied in codewords and emojis @xcite , and microaggressions that subtly expresses a prejudice attitude toward certain groups @xcite ; and (5) subjectivity measures annotation bias @xcite and manage annotator subjectivity involved in labeling various types of toxicity, which arises from differences in social and cultural backgrounds @xcite . in this paper, we analyze the toxicity of synchronous conversations in terms of the aforementioned dimensions by identifying explanation of toxicity as a form of norm categories (explainability), finding targets of hib words (target identification), leveraging context for both annotation and modeling (context sensitivity), asking annotators for implied knowledge statement (implication), and examining how human decisions align with machine decisions under different amounts of information (subjectivity). ## conclusion in this paper, we analyzed messages flagged by human moderators on twitch to understand the nature of norm violations in live-stream chats, a previously overlooked domain. we annotated 4,583 moderated chats from live streams with their norm violation category and contrasted them with those from asynchronous platforms. we shed light on the unique characteristics of live-stream chats and showed that models trained with existing data sets perform poorly in detecting toxic messages in our data, which motivates the development of specialized approaches for the synchronous setting. our experiments established that selecting relevant context is an important feature for detecting norm violations in the synchronous domain. we hope our work will help develop tools that enable human moderators to efficiently moderate problematic comments in real-time synchronous settings and make the user-experience in these communities more pleasant.