A Modular Taxonomy for Hate Speech Definitions and Its Impact on Zero-Shot LLM Classification Performance (2025)

Abstract: Detecting harmful content is a crucial task in the landscape of NLP applications for Social Good, with hate speech being one of its most dangerous forms. But what do we mean by hate speech, how can we define it and how does prompting different definitions of hate speech affect model performance? The contribution of this work is twofold. At the theoretical level, we address the ambiguity surrounding hate speech by collecting and analyzing existing definitions from the literature. We organize these definitions into a taxonomy of 14 conceptual elements—building blocks that capture different aspects of hate speech definitions, such as references to the target of hate. At the experimental level, we employ the collection of definitions in a systematic zero-shot evaluation of three LLMs, on three hate speech datasets representing different types of data (synthetic, human-in-the-loop, and real-world). We find that choosing different definitions, i.e., definitions with a different degree of specificity in terms of encoded elements, impacts model performance, but this effect is not consistent across all architectures.

## introduction in a world that is becoming increasingly onlinebased, detecting harmful content, specifically hate speech (hs), is crucial for maintaining the integrity of the democratic discourse and freedom of speech @xcite . the advent of large language models (llms) paved the way for a variety of new methods for detecting @xcite and countering hs @xcite , and for the creation of new artificial benchmarking data @xcite . in particular, novel methods for classifying harmful content diverge from conventional supervised learning that relies on input/output pairs, but uses only predetermined prompts without examples @xcite or adding further information on the task @xcite . a crucial role in refining prompts for zero-shot classification is played by the definition of the target construct, i.e., in the focus of this paper, the definition of hate speech. 1 as typical of social constructs of the social sciences, the definition of hs is ambiguous @xcite and cannot be easily framed in a static dimension. this is a relevant issue for the community, because it affects the interoperability of resources annotated at high cost, and the comparability of the results (and insights) drawn from their modeling, when, for example, different definitions are used for equivalent concepts @xcite . the contribution of our work is twofold: conceptual/theoretical and experimental. at the conceptual level, we contribute to structuring the conceptual landscape of hs by collecting and qualitatively organizing various definitions for hate speech. the goal of this analysis is to identify a set of conceptual elements (ces), i.e., the conceptual building blocks present in the definitions, which encode their key dimensions. for instance, all definitions highlight its problematic nature (@xmath0. however, only some definitions include potential consequences of hate speech (@xmath1. this taxonomy serves as a scaffold for constructing and analyzing definitions, which we believe is a novel and practically valuable contribution to both the nlp and social science communities. with these conceptual elements, we create a three-layer taxonomy (fig. 1 ) that we complement with a curated collection of definitions that arise from their combination. the collection of definitions can be seen as a structured, modular summary of the original set of definitions we reviewed and constitute a resource that we make available to the community for further experimentation. the second contribution is experimental: starting from the idea that llms already encode extensive knowledge due to their pre-training and instruction-tuning @xcite we employ our definitions from the collection to carry out zeroshot prompting experiments on three hate-speech datasets, representing different data types: hate-check @xcite , learning from the worst @xcite and measuring hate speech @xcite . we employ three different llms: llama-3, mistral and flan-t5. we conduct an in-depth error analysis and exploit the hatecheck fine-grained annotation regarding types of hate. 2 our results demonstrate the usefulness of our modular approach to build hate speech definitions as prompts for zero-shot classification. we find that varying construct definitions affects model performance, but this effect is not consistent across all model architectures and datasets. in some cases, more detailed definitions reduce false negatives, in others they primarily decrease false positives; more specifically, our error analysis shows that more detailed definitions improve performance in cases requiring nuanced distinctions between hate categories (i.e., implicit hate). similar strategies have been observed in hs detection. prompting llms with information on the task different from examples has improved performance in detecting hs @xcite . promising strategies involve prompting rationales or chain-of-thoughts alongside the task in zero-shot learning (zsl), few-shot learning, or fine-tuning @xcite . these approaches have shown that in-context learning, particularly in the context of zsl, is a worthwhile direction to explore @xcite . ## results as outlined in sec. 4.1, our experiments followed a two-step approach: llama-3 performs best without any definition (no) in two out of three datasets, suggesting potential data leakage from hatecheck. while in the lftw dataset, we encounter the only instance in which the best definition is the one of the dataset itself (own). among the crafted definitions, hsb + edt performs best for hatecheck and lftw, while hsb + edfoc + edpc is optimal for the mhs dataset. mistral achieves its highest scores with either no definition or offensive language (ol), implying an internalized concept of hate speech that aligns with offensive language. among crafted definitions, hsb + edfoc performs best in two datasets, while hsb is most effective in hatecheck. flan-t5, unlike the other models, benefits consistently from definition prompting. performance improves as definitions become more detailed, with hsb + edt yielding the highest results in hate-check and mhs, while the most extensive definition (hsb + edfoc + edpc + edt) is optimal for lftw. ## conclusion in this work, we explored the conceptualization of the construct definition of hs and its influence on zero-shot prompting on three datasets. our starting point has been the review of existing hs definitions, from which we inductively derived a set of conceptual elements. we then combined the different elements in the taxonomy to build a collection of definitions that lend themselves as prompts for llm modeling. thus, the taxonomy and the collection definition are not just a conceptual contribution of our work, but also a concrete resource that can and should be used by researchers to structure their operationalization of the hs construct, thereby contributing to a clearer research landscape. furthermore, the three-layers taxonomy, allows for combinations reflecting different levels of detail, which can be employed in annotation tasks in the descriptive vs. prescriptive paradigms @xcite . in our experiments, we exploited the definition collection for a series of zero-shot experiments, with the definitions serving as a series of curated prompts with increasing level of details. our results show that varying construct definitions affects model performance, in a complex constellation of patterns. some models benefit from detailed construct definitions by reducing false negatives, while others primarily decrease false positives. our micro-analysis of different hatecheck functionalities shows that incorporating specific conceptual elements targeting particular types of hate improves model performance, especially in cases requiring nuanced distinctions between hate categories. given our findings, we recommend that such a modular inspection of possible definitions should be employed for other complex constructs, beyond hs. moreover, our findings do have practical implications for the usage of llms in production. models that benefit from detailed construct definitions by reducing false negatives are, for example, better suited for high-recall moderation strategies, ensuring that fewer instances of hate speech go undetected. conversely, models that primarily lower false positives are more appropriate for high-precision approaches, minimizing the risk of over-flagging benign content. by strategically refining definitions and incorporating targeted conceptual elements, moderation systems can be optimized to balance recall and precision according to platform-specific goals.