Improving Hate Speech Classification with Cross-Taxonomy Dataset Integration (2025)

Abstract: Algorithmic hate speech detection faces significant challenges due to the diverse definitions and datasets used in research and practice. Social media platforms, legal frameworks, and institutions each apply distinct yet overlapping definitions, complicating classification efforts. This study addresses these challenges by demonstrating that existing datasets and taxonomies can be integrated into a unified model, enhancing prediction performance and reducing reliance on multiple specialized classifiers. The work introduces a universal taxonomy and a hate speech classifier capable of detecting a wide range of definitions within a single framework. Our approach is validated by combining two widely used but differently annotated datasets, showing improved classification performance on an independent test set. This work highlights the potential of dataset and taxonomy integration in advancing hate speech detection, increasing efficiency, and ensuring broader applicability across contexts.

## introduction research has shown a direct link between the rise of online hate speech and offline events @xcite , highlighting the growing impact of digital platforms on real-world occurrences. as of april 2023, there are an estimated 4.8 billion global social media users, making up about 59.9% of the world's population @xcite . this massive reach underscores the scale of the problem, with facebook alone removing 38.3 million instances of hate speech in the first three quarters of 2023 @xcite . these numbers emphasize both the urgency and magnitude of the issue, making it a top priority for the research community. the challenge lies in balancing the preservation of free speech with the need to protect individuals from harm. while algorithms play a key role in addressing this issue, they are just one part of a broader, multifaceted approach. in this context, this research aims to develop efficient and effective algorithmic solutions for hate speech detection. one main challenge in the field is that the understanding of hate speech varies and is influenced by factors such as topic @xcite , author @xcite , and time @xcite , among others. even within the legal context, it is a complex process deciding whether a statement should be classified as hateful or not. in response, research, private, and public entities have developed their own definitions and community standards, legal frameworks, or annotation guidelines @xcite . especially in the research field, the available datasets heavily depend on the annotation procedure and the definitions of hate speech provided to the annotators @xcite . this dependence and wide variety of definitions makes it challenging to compare @xcite or merge datasets annotated within different annotation schemas. while the field of available annotated hate speech corpora is limited to begin with, this additional limitation of incompatibility further complicates efforts to provide general and reliable hate speech detection. this research addresses this gap by providing a machine learning structure that combines existing definitions and datasets. it identifies mismatches in definitions, faults during the annotation combining process, and missing labels in datasets. the study demonstrates the feasibility of merging annotation schemas and datasets to detect a wider variety of hate speech definitions using just one trained classifier. it establishes that a single general taxonomy can be created and employed for multi-label federated training of a classifier, thereby improving prediction quality. the approach is evaluated using two standard research datasets and their respective definitions. the outcome involves the creation of a comprehensive hate speech taxonomy and the training of a general hate speech classifier. the scripts used for preprocessing, dataset construction, training, and evaluation are available as part of the paper. 1 this offers a deeper insight and facilitates the reproducibility of our work. please note that the used datasets have to be obtained from the cited sources. algorithmic detection -for detecting hate speech, toxic speech, abusive language, and related areas, the predominant algorithmic approach has utilized supervised transformer-based architectures @xcite @xcite . fine-tuning transformer models, particularly bert @xcite , has demonstrated significant performance enhancements compared to other methods @xcite @xcite . recently, the focus has shifted towards using pre-trained large language models combined with prompting techniques for hate speech detection @xcite @xcite . taxonomy and ontology matching -several researchers have aimed to create general hate speech ontologies @xcite and taxonomies @xcite @xcite . @xcite integrated their taxonomy into a transformer-based hate speech detection model, partially building on existing taxonomies and combining them to annotate a new dataset. the practice of merging ontologies is well established @xcite . however, no research has yet combined hate speech taxonomies to make existing datasets suitable for iterative federated learning. federated and continuous learning -federated learning for hate speech detection is crucial as it mitigates privacy concerns related to data sharing. a key development is @xcite , which introduces a binary hate speech classifier using a decentralized architecture, demonstrating superior performance across datasets while preserving privacy. another significant study, @xcite , explores multi-class federated learning on a static dataset with uniform annotations, disregarding annotation mismatches and emphasizing distributed training benefits. in continuous learning, @xcite propose a novel framework for detecting problematic content by integrating various datasets and treating each label as an independent classification task. this research directly builds upon the work of @xcite , @xcite , and @xcite . it extends the findings of @xcite and @xcite by demonstrating that federated training for hate speech detection is feasible not only for binary classification but also for multi-label hate speech datasets with varying definitions of hate speech. in relation to @xcite , it advances the research by integrating labels into a unified taxonomy with hierarchical aspects, introducing a deeper semantic relationship model, and showing that this model can be continuously adapted. ## methodology the research is divided into three main parts. first, a general hate speech taxonomy is created. second, this taxonomy is used to fine-tune a pre-trained multi-label hate speech detection model multiple times on two different datasets (see sections 4 and 6). lastly, continuous evaluation is conducted after each training cycle. each step is detailed in this section (see figure 1 ), with selected datasets, taxonomies, and the models serving as examples to demonstrate the approach's functionality. in this step, classes that cannot be merged are identified and removed. a word-level matching of annotations between the original and the new general taxonomy is introduced. the class hierarchy of the general taxonomy is represented through a one-hot encoded vector; when a subclass is flagged as identified, the parent classes must be present too. ## datasets two primary datasets with different annotations were selected for this research, along with two additional datasets: one for evaluation and one for balancing the two main datasets during training with non-hateful statements. the first main dataset, provided by @xcite , is a large, dynamically generated collection of 41,255 entries created over four rounds, with 54% of the entries being hateful. the dataset includes 11 english-language training datasets for hate and toxicity from hatespeechdata.com. its hierarchical taxonomy, based on robert c nickerson and muntermann (2013), classifies entries into hate and no-hate categories. the hate entries are further divided into five types (derogation, animosity, threatening language, support for hateful entities, dehumanization). additionally, 29 identities as hate targets are annotated. the annotations were performed by 20 trained annotators. the second main dataset compiled by fanton et al. ( 2021 ) is also a dynamically generated human-in-the-loop dataset, containing 5,000 hateful statements. created over two cycles with human input in between, the initial dataset included 880 statements and was developed in collaboration with 20 experts from various ngos. the annotations featured 10 labels ("disabled," "jews," "overweight," "lgbt+," "mus-lim," "women," "people of color," "ro-mani," "migrants," "other"). three trained students were involved in the annotation process. the dataset from fillies et al. (2023b) was selected for non-hateful statements, as only the hateful entries were selected from the two main datasets, and training a classifier solely on those would likely result in overfitting. this dataset, in english, includes annotated discord messages collected between march 2021 and june 2022, comprising 88,395 chat messages. around 6.42% of the messages were classified as hate speech. the final support dataset, from ljubešić et al. (2021), was chosen for validation and independent evaluation of the classifier's performance. it consists of youtube comments collected between january and may 2020, with approximately 50% hate and 50% non-hateful examples. the taxonomy from @xcite formed the basis for the merge due to its thoroughness. it initially distinguishes between hate and non-hate statements. hate types from @xcite were grouped under the label "types_of_hate," which was absent in @xcite . adjustments were made for hate targets, with seven out of 11 classes from @xcite fitting directly into the new taxonomy. the remaining classes, like "gender," "intersectional," and "disability," required modifications. due to @xcite introducing the labels "disabled" and "overweight", a class regarding physical attributes was introduced, also containing the label "gender," which then includes the class "gender minorities," unlike @xcite where it is independent. the last label from @xcite , "intersectional," was not included explicitly, as it is contained in the multi-label encodings (e.g., black women) that are represented in the taxonomy. the classes ("jews", "muslim", "women", "romani", "migrants") from @xcite were already covered in the taxonomy. the label "people of color" from @xcite was initially introduced as an independent label under the class "physical_attributes/skin_color/" next to the labels "black" and "white." however, the evaluation of the trained network's performance clearly showed this as a mistake, making it necessary to make "black" a subclass of "people of color." the main challenge was the label "lgbt+" by @xcite due to its covering of multiple aspects. it is first a political and social movement, standing for "lesbian, gay, bisexual, transgender, plus other sexual and gender identities," making it difficult to locate in the existing classes of gender and sexual orientation. the decision was made to include it in the taxonomy as a movement. it is noteworthy that in the actual dataset annotations by @xcite , labels appeared that were not represented in the provided taxonomy, such as "old.people," "russian," "lgbtq," "eastern.europe," and "non.white." these labels were included in the new general taxonomy with their own classes. however, the label "other" from @xcite had to be disregarded. the final taxonomy consists of 23 classes and 43 leaves, merging labels from both taxonomies directly or through abstraction. ## experimental classifier this section describes the creation of an experimental classifier. the classifier proves the validity of the concept as a proof-of-work. as detailed in the methodology section (3), the labels in the existing datasets from @xcite can be matched to the labels of the new taxonomy, creating a new annotation schema for the dataset. the annotated dataset is then used to fine-tune a pretrained language model to be a multi-label hate speech classifier. after this initial training, the classifier is used to reannotate the second dataset from @xcite , introducing the new annotation schema and providing insights into the created taxonomy, missing labels, and different underlying definitions of hate contained in the two datasets. the predicted annotations can then be merged with the existing human annotations and used to fine-tune the network again. if the approach holds merit, the minimum requirement is that the hate speech prediction quality of the network increases on an independent test set after the training cycles. this section describes the steps of this process. ## conclusion and outlook the results of this research demonstrate the feasibility of combining different hate speech taxonomies into a single, general taxonomy, which can be used to train a classifier capable of predicting a broader range of hate speech definitions. this approach reduces the need for multiple niche models, minimizing computational resources, and allows for model training without sharing sensitive data, thus addressing privacy concerns. the semantic relationships encoded in the labels also enhance generalizability for further training, aligning with current research in federated learning and continuous learning for hate speech detection. by iteratively fine-tuning a pre-trained multilabel classifier on two distinct datasets, the research shows that a general taxonomy can improve hate speech detection, leading to higher performance in classifying general hate speech, as demonstrated on an independent evaluation test set. this work serves as proof that a general taxonomy can be used in multi-label hate speech classification, integrating diverse datasets and definitions of hate speech. it also suggests that, in the future, only trained networks need to be exchanged, not the sensitive datasets, advancing federated hate speech detection. looking ahead, further research is needed to explore automatic matching of taxonomies on both logical and semantic levels, including detecting mismatches based on definitions. validation with a broader variety of hate taxonomies, and possibly the creation of a hate speech ontology, is essential. additionally, encoding structural knowledge through ontologies holds significant potential. further work is needed on bias mitigation and quality assurance in the context of hate speech detection.