A Unified Taxonomy of Harmful Content (2020)

Abstract: The ability to recognize harmful content within online communities has come into focus for researchers, engineers and policy makers seeking to protect users from abuse. While the number of datasets aiming to capture forms of abuse has grown in recent years, the community has not standardized around how various harmful behaviors are defined, creating challenges for reliable moderation, modeling and evaluation. As a step towards attaining shared understanding of how online abuse may be modeled, we synthesize the most common types of abuse described by industry, policy, community and health experts into a unified typology of harmful content, with detailed criteria and exceptions for each type of abuse.

## introduction content moderation, the practice of monitoring and reviewing user-generated content to ensure compliance with legal requirements, community guidelines, and user agreements, is important for creating safe and equitable online spaces. while traditional content moderation systems rely heavily on human reviewers who use a set of proprietary guidelines to determine if content is in violation of policy, the use of algorithmic approaches has become a part of moderation workflows in recent years. while not a full replacement for human content moderators, the use of ai promises to reduce trauma and cost incurred by purely human-centric workflows. as a result, the ability to recognize abusive content using data-driven approaches has attracted attention from researchers in the computational and social sciences. to study, model and measure systems designed to recognize online abuse, researchers typically create labelled datasets using crowdsourcing platforms or in-house annotators. while the number of research datasets continues to grow @xcite , the research community has not reached a consensus on how common abuse types are defined. despite the use of best practices that leverage multiple annotators, definitional ambiguity can lead to the creation of datasets of questionable consistency @xcite @xcite . furthermore, without thorough domain understanding, research datasets built to capture abusive content may be prone to unintended bias @xcite . together, these shortcomings create challenges for reliable modeling and study of abuse as it occurs in the real world. harmful content has also drawn attention from internet companies, such as those in the social media, online gaming, and dating industries, who seek to protect their users from abuse. these companies typically employ a trust and safety organization to define and enforce violations of content policies, and to develop tools aimed at identifying instances of abuse on their platforms. several online platforms which have seen large volumes of harmful content on their platforms, have created content policies that can be useful in specifying definitions of various abuse classes. in the absence of a standard upon which content policies can be based, community standards within digital platforms are largely shaped by users who report abuse they have experienced firsthand. additionally, some aspects of content policies are informed by requirements handed down from local law enforcement agencies wishing to prosecute users engaging in illegal activity online. recently, the demand for internet companies to more aggressively reduce the spread of cyberbullying, radicalization, deception, exploitation, and other forms of dangerous content has been increasingly called for by governmental and civil society organizations. the proposed online harms bill in the uk and amendments to section 230 in the united states call for stricter accountability, trans-parency, and regulations to be imposed on companies hosting user-generated content. civil society organizations have yielded numerous proposals for better describing types of harmful content online so that internet companies may better understand the nature and impact of such content. in this paper, we enumerate, consolidate and define the most common types of abuse described in content policies for several major online platforms and white papers from civil society organizations. we look for commonalities in both what types of abuse have been identified and how they are defined. our goal is to provide a unified typology of harmful content, with clear criteria and exceptions for each type of abuse. while hate speech and harassment have attracted attention from the natural language processing community in recent years, upon close study, we find that the domain of harmful content is broader than many may have realized. we hope this typology will benefit content moderation systems by: ## methodology to develop a unified typology of harmful content, we employed a grounded theory approach, in which we synthesized inputs from several sources: ## abuse class definitions using the data and guidelines described in section 3, we arrived at the typology depicted in figure 1 . in the remainder of this section, we describe each type in detail. within each section, we present the types in lexicographic order. ## conclusions and future work upon careful synthesis of content policies, human rights treaties and recommendations from experts in physical and psychological harm, we have presented a typology of harmful content along with a set of best practices for developing precise definitions of types. in the future, we plan to report on the impact of how the proposed definitions impact the quality of datasets and models built using them, and to share public datasets based on this typology that may be used by the research community. we have published the typology at https: //gitlab.com/sentropy-technologies/ typology-of-online-harm and encourage those who study online abuse to contribute.