Introducing CAD : the Contextual Abuse Dataset (2021)

Abstract: Online abuse can inflict harm on users and communities, making online spaces unsafe and toxic. Progress in automatically detecting and classifying abusive content is often held back by the lack of high quality and detailed datasets. We introduce a new dataset of primarily English Reddit entries which addresses several limitations of prior work. It (1) contains six conceptually distinct primary categories as well as secondary categories, (2) has labels annotated in the context of the conversation thread, (3) contains rationales and (4) uses an expert-driven group-adjudication process for high quality annotations. We report several baseline models to benchmark the work of future researchers. The annotated dataset, annotation guidelines, models and code are freely available.

## introduction social media platforms have enabled unprecedented connectivity, communication and interaction for their users. however, they often harbour harmful content such as abuse and hate, inflicting myriad harms on online users @xcite @xcite . automated techniques for detecting and classifying such content increasingly play an important role in moderating online spaces. detecting and classifying online abuse is a complex and nuanced task which, despite many advances in the power and availability of computational tools, has proven remarkably difficult @xcite @xcite . @xcite argued in a recent review, research has 'struggled to move beyond the most obvious tasks in abuse detection.' one of the biggest barriers to creating higher performing, more robust, nuanced and generalisable classification systems is the lack of clearly annotated, large and detailed training datasets. however, creating such datasets is time-consuming, complicated and expensive, and requires a mix of both social and computational expertise. we present a new annotated dataset of ∼25,000 reddit entries. it contains four innovations that address limitations of previous labelled abuse datasets. first, we present a taxonomy with six conceptually distinct primary categories (identity-directed, person-directed, affiliationdirected, counter speech, non-hateful slurs and neutral). we also provide salient subcategories, such as whether personal abuse is directed at a person in the conversation thread or to someone outside it. this taxonomy offers greater coverage and granularity of abuse than previous work. each entry can be assigned to multiple primary and/or secondary categories (section 3). second, we annotate content in context, by which we mean that each entry is annotated in the context of the conversational thread it is part of. every annotation has a label for whether contextual information was needed to make the annotation. to our knowledge, this is the first work on online abuse to incorporate a deep level of context. third, annotators provided rationales. for each entry they highlighted the part of the text which contains the abuse (and the relevant parts for counter speech and non-hateful slurs). fourth, we provide high quality annotations by using a team of trained annotators and a time-intensive discussion-based process, facilitated by experts, for adjudicating disagreements (section 4). this work addresses the need for granular and nuanced abusive content datasets, advancing efforts to create accurate, robust, and generalisable classification systems. we report several baseline models to benchmark the work of future researchers (section 5). the annotated dataset, annotation codebook and code have been made available. 1 a full description of the dataset is given in our data statement in the appendix @xcite . many taxonomies include fine-grained labels for complex subcategories of abuse. @xcite label implicit varieties of hate, including 'adjectival nominalization', 'distancing' and 'othering' language. @xcite label content for six subtypes of misogyny: discrediting, using stereotypes, objectifying, sexually harassing, threatening violence, dominating or derailing. @xcite provide annotations for which group is targeted and the linguistic action (i.e., dehumanizing, delegitimizing or aiming to inflict harm). they provide flags for aggressive-ness, offensiveness, irony and stereotypes. @xcite provide annotations for 'social frames' (i.e., biases and stereotypes) about groups. they provide labels for (a) offence (yes/no), (b) whether a group is targeted, and (c) whether the abuse is intentional. @xcite identify different interpersonal abuse, including toxicity, aggression and attacks. some taxonomies explicitly separate abuse from closely-related but non-abusive forms of online expression. this reflects social scientific insights which emphasize the importance, but also difficulty, of making such distinctions @xcite . @xcite al. (2020) annotation and data the quality of annotations for abusive datasets has been widely critiqued, and inter-rater agreement scores are often remarkably low. @xcite report an alpha of 0.45, @xcite kappas from @xmath0.37 for offence to @xmath1.54 for hate, @xcite report kappa of 0.15 in the "mmh150" dataset of hateful memes, and fortuna and nunes (2018) report a kappa of 0.17 for a text-only task. in a classification study of prejudice against east asia, @xcite find that 27% of classification errors are due to annotation mistakes. low agreement is partly because abuse is inherently ambiguous and subjective, and individuals can perceive the same content very differently @xcite . many abusive content datasets use crowdsourced annotations @xcite @xcite . they are cheap and scalable but can be low quality and are often ill-suited to complicated tasks @xcite . trained experts with clear guidelines are often preferable for ensuring consistency (vidgen and derczynski, 2020). whether experts-or crowdsourced annotators are used, a diverse pool is needed as annotators encode their biases, backgrounds and assumptions into their annotations @xcite . most datasets use a simple majority vote over annotations to determine the final labels. however, majority agreement does not guarantee that content is correctly labelled, especially for complex edge-cases. one option is to use a method that adjusts annotators' impact based on their quality, such as mace @xcite . however, this may not work well on the most ambiguous content. groupdecision making processes present a promising way of improving annotation quality. @xcite use a collaborative multi-stage process to label micro-aggression and @xcite use a similar process for labelling news articles. this ensures more oversight from experts and reflection by annotators on the difficult content. it also provides a feedback loop for annotators to learn from mistakes and improve. a well-established problem with abusive content datasets is that each bit of content is marked up individually, without taking into account any content that came before @xcite . this can lead to poor quality annotations when content is ambiguous or unclear without knowing the context. detection systems which do not account for context are likely to be less applicable in the real-world, where nearly all content appears in a certain context @xcite . @xcite systematically investigate the role of context in a dataset of wikipedia comments by providing annotators the 'parent' before showing them the 'child' entry. in one experiment at least 5% of the data was affected. in a study of twitter conversations @xcite label replies to tweets based on whether they 'agree' or 'disagree' with the original message. notwithstanding these studies, further work is needed to better understand the role of context and how abuse emerges within threads, as well as the challenges of detecting deeply contextual content. ## taxonomy we present a hierarchical taxonomy of abusive content, which comprises six primary categories and additional secondary categories. it builds on critical social scientific research @xcite @xcite , and addresses issues in previous taxonomies, including those provided by @xcite al. (2020), waseem et al. (2017), founta et al. (2018 ) and vidgen et al. (2019a). it offers greater coverage by including three conceptually distinct types of abusive content (identity-directed abuse, affiliationdirected abuse and person-directed abuse) as well as three types of non-abusive content (neutral, counter speech and non-hateful slurs). the tax-entry abusive identity-directed abuse derogation animosity threatening glorification dehumanization affiliation-directed abuse derogation animosity threatening glorification dehumanization person-directed abuse abuse to them abuse about them non-abusive non-hateful slurs counter speech against identity-directed abuse against affiliation-directed abuse against person-directed abuse neutral figure 1: primary and secondary categories. ## discussion and conclusion we have presented a detailed dataset for training abusive content classification systems. it incorporates relevant social scientific concepts, providing a more nuanced and robust way of characterisingand therefore detecting -abuse. we have also presented benchmark experiments, which show much room for improvement. our analyses indicate numerous areas to explore further, including creating systems which explicitly model the conversation threads to account for context. predictive methods could be applied to understand and forecast when a conversation is turning toxic, potentially enabling real-time moderation interventions. more powerful models could also be applied to better distinguish the primary categories and to begin classification of the secondary categories. this could be achieved by also using the images to classify the content, which we did not do. finally, we would also expect the rationales to be of considerable use in future experiments, both for classification and to understand the annotation process. the current work has several limitations. first, the class distribution is heavily skewed towards the neutral class and some abusive categories have low frequencies. this better reflects real-world prevalence of abuse but can limit the signals available for classification. second, inter-annotator agreement was in-line with other research in this domain but could still be improved further, especially with 'edge case' content.