Detecting Community Sensitive Norm Violations in Online Conversations (2021)

Abstract: Online platforms and communities establish their own norms that govern what behavior is acceptable within the community. Substantial effort in NLP has focused on identifying unacceptable behaviors and, recently, on forecasting them before they occur. However, these efforts have largely focused on toxicity as the sole form of community norm violation. Such focus has overlooked the much larger set of rules that moderators enforce. Here, we introduce a new dataset focusing on a more complete spectrum of community norms and their violations in the local conversational and global community contexts. We introduce a series of models that use this data to develop context- and community-sensitive norm violation detection, showing that these changes give high performance.

## introduction online communities establish their own norms of what is acceptable behavior @xcite @xcite . these norms run the gamut from no hate speech or no personal attacks to more idiosyncratic expectations of content formatting and content sharing @xcite . community moderators are responsible for identifying and removing rule-breaking content, regardless of whether users violate rules intentionally or unintentionally due to unfamiliarity with community norms. moderators of online communities often face a tough challenge of triaging the massive flow of content @xcite @xcite ; for example, over 2 billion comments were posted to reddit in just 2020. 2 moderators have looked to technology to help support their 1 dataset, code, and models are publicly available at https://github.com/chan0park/normvio . we introduce a new approach to contextsensitive automated content moderation that explicitly encodes community norms. using a new dataset of 51k conversations across 3.2k communities, we show that the most commonly-studied norm violation behavior in nlp, hate speech, corresponds to a small minority of cases in which moderators intervene in practice. we then create multiple models to detect when moderators intervene and why they intervene, adapting to the norms and rules of a community. our paper offers the following four contributions towards advancing the future of nlp in community and context-specific moderation. first, in a large scale analysis of rule and moderation behavior, we show that subreddits vary considerably in their rules, with only some common themes. however, in practice, most rules are not enforced and, further, the enforcement of some types of rules, e.g., incivility, is highly varied across communities. second, we introduce a new dataset, normvio, of 51k conversations across 3.2k subreddits and map the 25k rules from these communities into nine categories of context-specific unacceptable behavior, including five types of incivility. third, we introduce a new series of models aimed at detecting and explaining rule-violating behavior based on norms and rules of the community. our approach enables not only identifying that conversation in a particular community (with particular rules) is likely to violate a rule, but also which rule. we demonstrate the effectiveness of these models, showing our best model attains an f1 of 78.64 across all rule types, a 50% improvement over context-insensitive baselines. finally, we perform an in-depth analysis of how much conversation context and communitysensitivity affects predictability. our work points towards key challenges in detecting particular rule violations, while providing high accuracy in others, which can allow moderators to quickly intervene. more generally, our work provides a clear next step for nlp to look beyond one-size-fits-all methods for detecting incivility to developing holistic, context-sensitive approaches that better suit the needs of moderators and their communities. ## normvio dataset prior work has created datasets used to detect single types of norm violations in social media messages (e.g. incivility, hate speech or hostility) @xcite . however, these datasets typically focus on isolated texts and do not provide prior conversational context or community-specific details. in order to detect representative types of norm violations and account for context, we construct a new dataset-normvio-a collection of 52k english conversation threads on reddit. normvio includes comments removed for violating a variety of community norms beyond the traditional hate speech and incivility, such as spamming or violating community format/topics. furthermore, normvio provides additional context beyond the norm-violating comment itself with (a) the entire conversation thread (i.e., the original post and prior comments) and (b) the subreddit (i.e., community) in which the comment was posted. ## detecting and explaining community norm violations with normvio, we can now train models for detecting contextualized, fine-grained community norm violations. we present two tasks: (1) detecting community norm violations, and (2) explaining community norm violations. the former identifies coarse categories of norm violations detailed in §2, and the latter is aimed at identifying specific local community rules being violated, to facilitate moderation transparency. for each task, we compare model variants without or with varying types of incorporated context, including conversation history and community information (e.g. subreddit name). ## analysis how many violations do current systems miss? in part due to their targeted focus, the perspective and incivilhate baseline models miss a substantial proportion of the total norm violations. figure 6 shows the confusion matrices of the violation detection task, where labels are aggregated over all violation types to test how many violations overall are not captured by these systems. the results show that perspective and incivilhate fail to recognize 42% and 34% of all violations, respectively. moderators on platforms like reddit must triage huge numbers of comments daily and this points to a clear gap between current practice (represented by the baselines) and indicates what moderators act on in practice. ## conclusion online communities establish their own norms for what is acceptable behavior. however, current nlp methods for identifying unacceptable behavior have largely overlooked the context in which comments are made, and, moreover, have focused on a relatively small set of unacceptable behaviors such as incivility. in this work, we introduce a new dataset, normvio, of 51k conversations grounded with community-specific judgements of which rule is violated. using this data, we develop new models for detecting context-sensitive rule violations, demonstrating that across nine categories of rules, by incorporating community and conversation history as context, our best model provides a nearly 50% improvement over context-insensitive baselines; further, we show that using our models, we can explain which rule is violated, providing a key assistive technology for helping moderators identify content not appropriate to their specific community and better communicate to users why. our work enables a critical new direction for nlp to develop holistic, context-sensitive approaches that support the needs of moderators and communities.