A Disaggregated Dataset on E nglish Offensiveness Containing Spans (2025)

Abstract: Toxicity labels at sub-document granularity and disaggregated labels lead to more nuanced and personalized toxicity classification and facilitate analysis. We re-annotate a subset of 1983 posts of the Jigsaw Toxic Comment Classification Challenge and provide disaggregated toxicity labels and spans that identify inappropriate language and targets of toxic statements. Manual analysis shows that five annotations per instance effectively capture meaningful disagreement patterns and allow for finer distinctions between genuine disagreement and that arising from annotation error or inconsistency. Our main findings are: (1) Disagreement often stems from divergent interpretations of edge-case toxicity (2) Disagreement is especially high in cases of toxic statements involving non-human targets (3) Disagreement on whether a passage consists of inappropriate language occurs not only on inherently questionable terms, but also on words that may be inappropriate in specific contexts while remaining acceptable in others (4) Transformer-based models effectively learn from aggregated data that reduces false negative classifications by being more sensitive towards minority opinions for posts to be toxic. We publish the new annotations under the CC BY 4.0 license.

## introduction the amount of toxic content on the internet is increasing and causes harm. especially implicit offensiveness still often goes undetected @xcite . given the absence of widely accepted definitional distinctions between the terms offensiveness and toxicity @xcite , we use them interchangeably. creating effective automated content moderation systems requires two key elements: nuanced understanding of online norm violations and incorporation of diverse opinions on what content warrants moderation. the need for perspectivist offensiveness detection perceptions of what content is harmful depend on individual, contextual, and geographical factors @xcite @xcite . researchers report disagreements in the annotation of toxicity explained by previous annotations by the same annotator @xcite , sociodemographics (e.g. @xcite beliefs @xcite , and moral values and geocultural factors @xcite . the human perception of what constitutes harmful language is inherently reflected in the classifiers trained on human-labeled data to identify toxic speech. if perceptual differences in what makes a remark toxic are not taken into account when training these systems, what is considered toxic may be disproportionately influenced by the societal majority or, in the worstcase scenario, by an arbitrary group of annotators. therefore, a one-size-fits-all approach to content moderation is unable to account for the diverse needs of different users @xcite @xcite . perspectivism in machine learning (ml) represents a paradigm shift from consensus-based labeling to embracing annotator diversity. for each data instance, multiple labels are collected and, where possible, maintained as disaggregated annotations alongside annotator metadata throughout the ml pipeline @xcite . this paradigm aligns with descriptive annotation approaches, which embrace annotator subjectivity as a meaningful signal rather than noise. descriptive annotation enables the modeling of diverse beliefs and perspectives, contrasting with prescriptive annotation that enforces uniform interpretation through strict guidelines @xcite . the need for offensiveness annotations beyond classes in (semi-) automated content moderation, explainability contributes to a greater understanding and trust of users @xcite and content moderators @xcite . experienced moderators work more efficiently when provided with structured explanations that pinpoint harmful content and articulate why it violates community standards @xcite . in text annotation, a span refers to a contiguous sequence of tokens within a document that is marked and labeled. rather than annotating entire documents or sentences, spans allow annotators to identify and categorize specific portions of text. annotators can pinpoint exactly which parts of a text exhibit the phenomenon of interest. text classification models can learn from the specific linguistic features within marked spans, leading to more accurate predictions about similar text segments. different spans within the same text can receive different labels, capturing the complexity of real-world documents where multiple phenomena may coexist, facilitating error analysis, model debugging, and model explainability @xcite . main contributions by re-annotating 1,983 comments from the jigsaw toxic comment classification dataset @xcite , 1,561 of which received annotations from four or five annotators, we classify toxic utterances at the post level and we identify spans comprising the targets of toxic utterances, and spans comprising vulgar expressions while maintaining disaggregated labels from multiple annotators. we find that high disagreement among five annotators meaningfully signals cases where subjective elements influence the perception of toxic statements. our analysis demonstrates that disagreements arise from divergent interpretations of borderline toxic content. we find substantial disagreement on toxicity classifications involving non-human targets. further, when evaluating whether a span consists of inappropriate language, disagreement occurs not only on inherently questionable terms, but also on words that may be inappropriate in specific contexts while remaining acceptable in others. lastly, while annotators generally recognize targets of toxic language, repeated target mentions within comments can pose hurdles for human annotators and for ml approaches to extracting spans comprising targets. despite differences in definitions and annota-tion approaches, we find broad agreement between the jigsaw annotations and our own. additionally, experiments show that transformer-based models effectively learn from aggregated data, which reduces false negative classifications by being more sensitive towards minority opinions for posts to be toxic. we release the new annotations under cc by 4.0 licensing with the underlying comment text being governed by wikipedia's cc-sa-3.0 foot_0 . ## related work recent advances in toxicity annotation include the development of granular labeling frameworks, publication of disaggregated annotation datasets, and diversification of annotator demographics. ## data source we source the data from the toxic comment classification challenge from jigsaw. it contains wikipedia comments which have been labeled by human raters for toxic behavior. annotated labels in the dataset are toxic, severe toxic, obscene, threat, insult, identity hate. the data is published under the cc0 license, with the underlying comment text being governed by wikipedia's cc-sa-3.0. we source 1700 not toxic comments. additionally, since we are interested in nuanced toxicity cases, we select comments labeled as toxic without any additional toxic categories and sample 1700 of these. we further select insults without any additional toxic categories, excluding comments with more severe or multiple toxic labels. from this pool of data, 1983 posts are annotated. figure 1 shows the distribution of comment lengths in our dataset, excluding outliers. ## annotation schema we adopt the annotation schema used for the german austrotox dataset @xcite , making the two datasets containing different data sources and cohorts of annotators compatible and allowing for multilingual analyses. observe the annotation strategy in figure 2 . we classify each comment as insult, incite to hate or violence or not offensive. since we do not source for incites to hate or violence in the jigsaw dataset, the number of posts labeled as incite to hate or violence in our dataset is limited. therefore, we create an offensiveness / toxicity class by merging classes insult and incite to hate or violence. for non-offensive and offensive comments, we annotate vulgarities since both offensive and non-offensive posts can contain vulgarities. for offensive posts, we additionally annotate the targets of the offensive statement and the type of target. if the target is only mentioned via a pronoun, we annotate the pronoun as the target. adopting a definition of vulgarity similar to that employed by @xcite , we use the following definitions for classes and spans: insult an insult pursues the recognisable goal of disparaging the addressee or the object of reference. incite to hate or violence an incite to hate or violence against a person or a group of people. it is often hard to draw the line between insults and incites to hate, as insults always somewhat incite hate. for this annotation task, we define insults to be less severe than incites to hate or violence. offensive / toxic an insult or an incitement to hate or violence. not offensive / not toxic not an insult nor an incite to hate or violence. vulgarity rude, obscene, foul or boorish language that is inappropriate for civilized discourse. target group the target of an offensive post is a group of persons or an individual insulted based on shared group characteristics. target individual the target of an offensive post is a single person, not insulted based on shared group characteristics. target other the target of an offensive post is not a person or a group of people. we position our work within the descriptive annotation paradigm, recognizing that toxicity perception contains inherent subjective elements @xcite . determining whether someone intends to disparage a target, distinguishing between hate incitement and mere insults, and identifying when comments cross into inciting violence all involve subjective judgment calls. similarly, the threshold for what constitutes inappropriate passages in civilized conversation varies across readers and contexts. among our annotation tasks, identifying the target of offensive statements and categorizing the target type represents the most objectively answerable component. our definition of vulgar language is deliberately expansive, extending beyond conventional sexual, scatological, and religious profanity to include any language inappropriate for civilized discourse. we recognize that determinations of vulgarity are both contextdependent and inherently subjective, as language deemed acceptable in casual forums may prove inappropriate in structured, goal-oriented discussions and edge-case acceptability varies by reader. ## annotation campaign we conduct the annotation with master's students in data science. thirty percent of annotators are registered as female in this course, though this institutional designation may not reflect their actual gender identity. the majority of annotators are between 19 and 26 years old, with all demonstrating at least b2-level english proficiency. most annotators originate from eastern europe. we acknowledge the importance of representing the perspectives of a population by including diversity in the annotator pool (clemmensen and kjaersgaard, 2022). our annotator pool's homogeneous sociodemographics, constrained by resources and participants in the course, present a limitation in our data. on the other hand, the demographic composition in our dataset contrasts with typical nlp annotation practices, where annotators are predominantly recruited from english-speaking countries. however, english-language online discussions reach global audiences with diverse cultural backgrounds and perspectives on toxicity. we therefore argue that our annotator pool provides valuable demographic diversity to the current landscape of toxicity datasets. furthermore, opinions on what constitutes toxicity are influenced by a multitude of factors beyond just sociodemographics. while we do not explicitly capture annotator characteristics for individual annotators, these factors are implicitly reflected in the disaggregated annotations @xcite . consequently, we view our dataset as a valuable addition to the broader collection of resources, capturing user perspectives in various ways. the annotation campaign was reviewed by the ethics committee of the first author's institution. each annotator annotates about 200 comments, which takes approximately 1.5 to two hours. the dataset contains a higher proportion of offensive comments than the typical distribution in a user forum, but we only source comments with labels toxic or insult and exclude more severe labels. the annotators are explicitly informed that they have the option to cease annotation if they feel overwhelmed by the task without facing consequences, and about the publication of the data, and they receive comprehensive compensation through course credits for their efforts. figure 3 shows the number of annotators per comment. the vast majority of posts is annotated by five annotators. ## disagreements in the annotations we calculate inter-annotator agreement using krippendorff's alpha across all annotation categories and conduct manual analysis to identify factors correlating with disagreement at both the post and span levels. ## disagreement between the jigsaw dataset and our dataset to enable meaningful comparisons with both the jigsaw dataset annotations and the german aus-trotox @xcite dataset using a different data source but shared annotation framework, we aggregate the data using an approach that reduces false negative classifications by being more sensitive towards minority opinions for posts to be toxic. ## classification experiments we conduct experiments on the aggregated data in order to show that the labels provide learnable signals. we conduct experiments on binary offensiveness classification, token classification of vulgar passages, and passages constituting the different types of targets. we fine-tune and evaluate encoder-based models, and we evaluate the few-shot performance of decoder-based models in a 10-fold cross-validation setting. we fine-tune encoder-based models on all three tasks independently. this means that the target detection task inherently includes offensiveness classification, as we only annotate targets of offensive statements. we choose electra large foot_1 @xcite and roberta large foot_2 @xcite for our experiments, as they exhibit good performance at the semeval-2023 task 10: explainable detection of online sexism @xcite . additionally, we assess the in-context learning performance of the following large language models: gpt 3.5(gpt-3.5-turbo-1106) @xcite , gpt 4 (gpt-4-1106-preview) @xcite , llama 3 foot_3 (ai@meta, 2024) and mistral foot_4 @xcite . we use the same prompts as pachinger et al. 2024. they contain an offensiveness definition, the post to be classified, and for the five-shot scenario, randomly sampled annotated example posts. due to limited performance for mistral and llama3, we adjust the prompt, requiring them to respond with only 0 or 1, and we define the token with the higher logit as the llama3 and mistral's prediction. we tokenize the spans generated by the generative models with the roberta tokenizer. we compute the micro f1 by adding up the values of the confusion matrix for the three target classes using nakayama's (2018) framework. results in line with the results of the experiments on the austrotox dataset, we find that especially the vulgar token detection task profits from fine-tuning. none of the models achieve good performance on target token detection. however, several factors explain these poor results. first, this is a challenging four-class classification task where the evaluation using the micro-f1 score focuses only on target classes, excluding the predominant non-target class. this evaluation approach, combined with the sparse distribution of target spans in the data, inherently produces lower scores compared to the other two tasks. the high level of human annotator disagreement provides additional insight into these performance issues. targets frequently appear multiple times within posts and are often referenced only through pronouns, creating ambiguity. given that human annotators struggled with the task, the poor model performance becomes more understandable. ## conclusion we re-annotate posts from the jigsaw toxic comment classification challenge, providing disaggre-gated toxicity labels and spans that identify inappropriate language and targets. this sub-document granularity enables more nuanced and personalized toxicity classification. manual analysis demonstrates that five annotations per instance effectively distinguish meaningful disagreement from annotation inconsistencies. we find high levels of disagreement on borderline toxicity cases, particularly for toxic statements targeting non-human entities. additionally, when annotating spans comprising inappropriate language, disagreement occurs both on inherently questionable terms and on contextsensitive words that may be acceptable in some settings but inappropriate in others. finally, experiments show that transformer-based models effectively learn from aggregated data that reduces false negative classifications by being more sensitive towards minority opinions for posts to be toxic.