Web(er) of Hate: A Survey on How Hate Speech Is Typed (2025)

Abstract: The curation of hate speech datasets involves complex design decisions that balance competing priorities. This paper critically examines these methodological choices in a diverse range of datasets, highlighting common themes and practices, and their implications for dataset reliability. Drawing on Max Weber’s notion of ideal types, we argue for a reflexive approach in dataset creation, urging researchers to acknowledge their own value judgments during dataset construction, fostering transparency and methodological rigour.

## introduction researchers in computer science, particularly within the nlp community, are increasingly devoting attention to online hate speech. as a deeply social phenomenon, online hate speech has been recognised in prior research for its potential to incite and propagate offline violence @xcite . since @xcite , there have been a plethora of hate speech datasets 1 with great diversity in their curation processes despite sharing the overarching goal of advancing state-of-the-art hate speech detection. as noted by previous research, this heterogeneity negatively affects crossdataset and cross-domain generalisation @xcite . at the same time, it has opened up other research directions, such as transfer learning @xcite . while the differences in datasets are highlighted in past survey studies @xcite , areas such as design goal and quality assurance are often overlooked. in this paper, we draw on max weber's notion of "ideal types" @xcite @xcite (see §2) to highlight that the diversity in hate speech datasets are natural and unavoidable. instead of pursuing definitional completeness, researchers should adopt a reflexive dataset curation approach. we argue that a fully accurate and comprehensive decomposition of hate speech might not exist. instead, to progress as a field, the complexities of hate speech should be recognised and the perspectives and assumptions of researchers documented. we aim to answer the following research question: after deciding to curate a labelled corpus for hate speech detection, how has past research defined hate speech and how do the design decisions differ? in doing so, we make the following contributions: we provide an overview of weber's ideal types ( §2) and previous surveys ( §3). paper selection is outlined in §4. in §5, we outline key insights and observations. our discussion ( §6) syn-thesises and interprets our findings. the appendix includes breakdowns of the datasets analysed. ## weber's ideal types the inherent subjectivity and the variability in defining hate speech have been discussed within the nlp community @xcite @xcite . this subjectiveness makes hate speech detection as a classification task difficult. in discussing the subjectivity of hate speech detection, @xcite outline two contrasting paradigms to encourage researchers to either embrace or limit the subjectivity of the task to the fullest extent. cercas @xcite call for a separation between -isms and offence and distinguish individual differences from subjectivity. ideal types, conceived by the german sociologist max weber, are analytical heuristics that serve to make sense of complex social phenomena. they are not perfectly all-encompassing, nor do they represent the average. rather, in an observer's attempts to understand phenomena such as capitalism @xcite or, more relevant to this discussion, hate speech, these ideal constructs are created to "sort out" the underlying complexities. it is therefore inevitable that these constructs depend on the observer's frame of reference, and as a result the observer-whether consciously or unconsciouslyarticulates certain aspects that they deem worthy while suppressing those of less importance. viewed through a weberian lens, the subjectivity and variation of hate speech datasets are grounded in the frame of reference (cultural norms, historical perspectives, laws, moderation guidelines, and values) that actors (researchers from computer science, linguistics, gender/political/religious studies, criminology or law, annotators, platforms, moderators, speakers, recipients, bystanders, and counterspeech campaigners) choose to adopt and accept. prescriptive guidelines can limit variation @xcite , but may still introduce bias through the identity and values of the moderator, speaker, and recipient. weber names four ideal types of social action: foot_1 goal-rational (zweckrational): motivated by precise and strategic calculation with the aim of achieving some goals. value-rational (wertrational): motivated by values and beliefs despite their potentially sub-optimal consequences. affectual (affektuell): driven by emotions. traditional (traditional): based on established traditions and habits. in the context of hate speech, goal-rationality might see hate speech being used strategically to achieve political or ideological goals. researchers might be interested in how such discourse polarises public opinions and even radicalises the public to the extremes. from a value-rational perspective, hate speech might be expressed in ways that align with the speaker's beliefs about race, gender, or religion. the evaluation of such belief-driven hate speech is heavily dependent on whether the observer (e.g. a researcher, moderator, annotator, or a set of annotation guidelines) shares those values. affectual action hate speech can be an emotional response, such as anger or frustration. this category is relevant when considering hate speech in interpersonal conflicts such as wikipedia or code repository edit comments. moderators might struggle with distinguishing these reactionary expressions of emotions from more systematic hate speech. finally, traditional forms of hate speech are embedded in cultural and societal norms and traditions, such as casual misogyny or transphobia in some communities. this, too, requires the observer to be aware of their tradition and how it might affect their judgement of hate. by operationalising their concept of hate speech, researchers risk missing aspects of discourse that do not fit neatly with their ideal type. for example, anti-semitic conspiracy theories often do not contain explicit slurs but rely on coded language and misinformation (e.g. accusations of global control) (rathje, 2021). these types of covert, goaldriven hate have been overlooked by previous ideal types of hate speech. at the same time, however, it is unrealistic and perhaps impossible to create a perfect representation of hate speech. researchers must rely on using ideal types to study the areas in focus, and any ideal type is an idealised representation, bound to overlook certain aspects. actors use frames of reference to construct an ideal type. goal-rational actions, such as online moderation, may use prescribed guidelines. however, these are not stable, and the terms of reference can change over time and place. meta and x (formerly twitter) have changed their policies regarding transphobic hate speech. this highlights the challenge of developing prescriptive guidelines that remain relevant and applicable. by recognising that any operationalisation of hate speech is an ideal-typical construct, we argue no single decomposition can fully encapsulate the complexity of hate speech. instead, researchers should explicitly document their perspectives and assumptions, acknowledging the underlying subjectivities in their operationalisation. while their work provides valuable insights into identity and geo-cultural representation, our study takes a broader approach by examining the entire dataset curation process, including definitions, intended goals, and design choices. the biases revealed by @xcite and @xcite illustrate the gap between curators' ideal types-as conceptualised in their definitions and frameworks-and the realities of their final datasets, reinforcing our argument that dataset validity hinges on alignment with intended objectives rather than definitional completeness. ## selection criteria the primary source of our datasets is the community-maintained hate speech dataset catalogue 3 @xcite , which lists 3 hatespeechdata.com 124 research papers and their associated datasets across 25 languages but has limited coverage post-2023. to supplement this, we conducted a google scholar search paying particular attention to two venues. specifically, we conducted two targeted searches and one general search using the following query: we chose these keywords to broadly cover terms commonly used in existing literature. while we acknowledge scope-specific keywords such as "racism" and "sexism", we did not include those to avoid biasing the search towards specific types of hate. to target acl (association for computational linguistics) and acm (association for computing machinery), we suffix site:aclanthology.org and site:acm.org to the query respectively. for general search, we append their negative filters to reduce redundancy. we filter results to studies published from 2023 onward, considering only the first three pages of search results. we only select studies that introduce and describe a new dataset. non-textualcontent-based prediction (e.g. predicting using metadata, casavantes et al., 2023) are excluded, but re-labelled datasets are included along with their originals. 4 we verify consistency across multiple top-level domains (.com, .co.uk, .jp, and .hk). the search is conducted in incognito mode to remove any potential search engine personalisation. we do not conduct a full snowballing process due to its bias toward older studies and limited added value beyond our combined search strategy. we treat substantially different datasets introduced within the same paper as distinct datasets (e.g. @xcite , as the datasets differ in both data sources and collection methods. in contrast, we regard ethos @xcite as a single dataset despite its use of two data sources, since other aspects of its creation process remain consistent. in total, we retrieved 135 distinct datasets across 36 languages. figure 1 shows a breakdown of the number of datasets published in each year, split by source. 5 key insights and observations ## discussions a reflexive approach as hate speech detection inherently involves value judgements, it is crucial for researchers to adopt a reflexive approach throughout the dataset curation process, where the ideal types of hate and curatorial stances are critically examined and reported. in a prescriptive paradigm where disagreements and subjectivity are discouraged, the frame of references of the researchers can still shape their ideal-typical conceptualisation of the categories and definitions. therefore, researchers must critically examine and document their own value judgements and frame of references as these ultimately shape the annotated datasets and trained models. by making these aspects explicit, researchers can promote trans-parency and allow for a more nuanced understanding of goal-driven ideal-typical constructs. annotator composition we note the interplay between annotator composition and the author's ideal-typical conceptualisations. datasets with smaller, hand-picked annotator pools can more easily enforce a uniform ideal type through targeted training and discussions. this approach is more suited for prescriptive guidelines. conversely, crowdsourced datasets can capture greater diversity, aligning better with descriptive guidelines. however, the persistent underreporting of crowdsourcing annotator demographics presents a challenge in assessing the diversity of captured opinions. annotation aggregation while many datasets rely on majority voting, this method relies on two key assumptions: 1) ground truth is both obtainable and desirable, and 2) annotator consensus reflects this ground truth. whether these assumptions hold depends on the operational framework. in a descriptive paradigm, aggregating annotations removes the diversity of responses rather than captures it. additionally, majority voting leaves the underlying sources of disagreement unexamined, further introducing noise. alternative approaches such as moderation meetings provide a more robust approach for resolving disagreements but are underutilised. furthermore, datasets that discard instances with disagreement risk removing ambiguous cases, leading to an oversimplification of the task, which may reinforce existing biases. ## conclusion through a weberian lens, we examine hate speech datasets through max weber's ideal types of social action to understand the socio-political underpinnings. we illustrate examples of goal-rational hate, where political figures use hate and abusive language to mobilise the public for political gain, and value-rational hate, where hate speech is driven by ideological beliefs. moreover, affectual hate can be attacks driven by emotions such as frustration and anger, while traditional hate speech is often normalised and implicit. these ideal types offer a theoretical grounding to the operationalisation of hate speech while acknowledging the diversity of design choices of researchers. our analysis highlights how dataset construction is shaped by various factors, including the researchers' frame of reference and goal, which in turn influence key design decisions. we advocate for a reflexive approach to dataset construction in which researchers critically examine their own assumptions, operationalisation choices, and the socio-political contexts that shape their work.