Towards Fairness Assessment of D utch Hate Speech Detection (2025)

Abstract: Numerous studies have proposed computational methods to detect hate speech online, yet most focus on the English language and emphasize model development. In this study, we evaluate the counterfactual fairness of hate speech detection models in the Dutch language, specifically examining the performance and fairness of transformer-based models.We make the following key contributions. First, we curate a list of Dutch Social Group Terms that reflect social context. Second, we generate counterfactual data for Dutch hate speech using LLMs and established strategies like Manual Group Substitution (MGS) and Sentence Log-Likelihood (SLL). Through qualitative evaluation, we highlight the challenges of generating realistic counterfactuals, particularly with Dutch grammar and contextual coherence. Third, we fine-tune baseline transformer-based models with counterfactual data and evaluate their performance in detecting hate speech. Fourth, we assess the fairness of these models using Counterfactual Token Fairness (CTF) and group fairness metrics, including equality of odds and demographic parity. Our analysis shows that models perform better in terms of hate speech detection, average counterfactual fairness and group fairness. This work addresses a significant gap in the literature on counterfactual fairness for hate speech detection in Dutch and provides practical insights and recommendations for improving both model performance and fairness.

## introduction while the ease of expressing oneself on social media platforms has led to creative and meaningful interactions, it has also amplified the spread of hate speech -particularly content targeting specific groups based on ethnicity, gender, sexual orientation, and similar characteristics. to address this issue, researchers have developed numerous computational methods for detecting hate speech @xcite @xcite . however, the majority of these efforts focus on the english language. regardless of language, a critical question remains: are these detection models fair? in other words, do they detect hate speech targeted at all social groups with similar accuracy? if not, the models are unfair @xcite . unfair models risk perpetuating biases, which can exacerbate existing issues and erode users' trust in social media platforms. this work evaluates the fairness of hate speech detection models for the dutch language. consider the following two sentences: s1:"all moroccans are troublemakers." and s2:"all dutch are troublemakers". a fair hate speech detection model should classify both sentences as equally hateful since they share the same structure and level of negativity. however, if the model predicts s1 to be 98% likely to be hateful but s2 only 10%, this disparity indicates unfairness. this kind of bias often arises when sensitive identity attributes, such as nationality, ethnicity, sexuality, or religion, disproportionately influence model predictions @xcite . this example highlights the importance of counterfactual fairness: the principle that a model's decision should remain consistent if sensitive attributes in the input data are changed. for instance, s2 is the counterfactual version of s1 @xcite . if a model evaluates these sentences differently, it fails to meet the standard of counterfactual fairness @xcite . this challenge is particularly critical in hate speech detection, where sensitive attributes often appear in potentially biased contexts. more formally, a model m is considered counterfactually fair if it produces the same predictions for all possible values of a sensitive attribute a. mathematically, this can be expressed as: p r( ŷ | @xmath0, where @xmath1, a 2 } represents different values of the sensitive attribute. despite progress in counterfactual fairness research for the english language, not much work has been done in european languages such as dutch. several studies on dutch hate speech detection have been conducted using pre-trained language models and mainly focused on creating datasets and models for hate speech detection @xcite @xcite @xcite @xcite . this work aims to bridge the gap in the existing literature by exploring the counterfactual fairness of pre-trained models in detecting hate speech in dutch on social media platforms. we aim to answer the following research questions: what are the methods to generate counterfactual data for the dutch language? are hate speech detection models in the dutch language counterfactually fair? what is the impact of generated counterfactual data on the performance and fairness of the hate detection model in the dutch language? figure 1 presents the key steps in our methodology. we begin by curating a list of dutch social group terms and generating counterfactual sentences. to do this, we use large language models (llms) and other techniques like manual group substitution (mgs) and sentence log-likelihood (sll) originally proposed for english. through qualitative evaluation, we identify challenges in generating realistic counterfactual sentences that conform to the rules of dutch grammar. next, we fine-tune transformer-based hate speech detection models using the generated counterfactual data. we then evaluate the performance of these models in detecting hate speech in dutch. finally, we assess counterfactual fairness using counterfactual token fairness (ctf). we also compute group fairness metrics, namely, equality of odds and demographic parity. ## related work hate speech detection has been extensively studied @xcite @xcite @xcite @xcite . defining hate speech is inherently challenging, as it is a complex phenomenon influenced by interpretation @xcite . fortuna et al. @xcite proposed a comprehensive definition of hate speech: "hate speech is language that attacks or diminishes, incites violence or hate against groups based on specific characteristics such as physical appearance, religion, descent, national or ethnic origin, sexual orientation, gender identity, or others. it can manifest in various linguistic styles, including subtle forms or even through humour." this definition captures the diverse ways in which hateful language can be expressed. although most of the research in this domain has focused primarily on the english language, some studies have explored hate speech detection in multiple languages @xcite . notable examples include investigations into italian @xcite , danish @xcite , and spanish (plaza-del arco et al., 2021), among others. we focus on hate speech detection in the dutch language. among earlier works, tulkens et al. @xcite performed a dictionary-based approach for the detection of racist discourse in dutch using automated means. markov et al. @xcite explored features based on emotions and style for cross-domain hate speech detection in multiple languages including dutch. caselli et al. @xcite introduced a new dataset, dutch abusive language corpus (dalc v1.0), which comprises manually annotated tweets for abusive language. ruitenbeek et al. @xcite curated a corpus containing more than 11k posts on twitter in dutch which are abusive and offensive. hilte et al. @xcite investigated the demographics of authors who spread hate speech in dutch and found that older men indulge in more hate speech. vries @xcite used a bert-cnn based model for detecting the targets against whom hate is triggered on the x platform in dutch. however, none of these works address the important issue of fairness of dutch hate speech detection models. fairness is becoming increasingly important in the context of hate speech classification. a model is considered fair when it (1) does not use sensitive attributes in making decisions and (2) treats individuals with the same sensitive attributes similarly @xcite . approaches to increase fairness are predominantly based on sensitive attributes that point to (un)privileged groups, which are disproportionately likely to be positively classified as hateful by a model @xcite . although different notions of fairness exist @xcite , we focus on causality-based notions because evaluating causal relationships provides a more comprehensive evaluation of model fairness that can uncover model bias. more specifically, we apply counterfactual fairness @xcite which is defined as "the intuition that a decision is fair towards an individual if it is the same in (a) the actual world and (b) a counterfactual world where the individual belonged to a different demographic group". in the context of fairness of hate speech models @xcite , this means that changing the sensitive attribute in a sentence should not impact the outcome of classification. the sentences "some people are gay" and "some people are straight" should, therefore, receive a similar toxicity prediction by fair models. while most previous work focuses on english, we focus on the dutch language and evaluate hate speech detection models from a fairness perspective. ## counterfactual data generation evaluating the counterfactual fairness of a model requires access to counterfactual sentences. counterfactual generation is a data augmentation strategy that creates such sentences by modifying sensitive identity terms while preserving the original meaning. this additional data is then used to finetune the model with the goal of improving both performance and fairness. we apply three methods for generating counterfactuals: large language model (llm), sentence log-likelihood (sll), and manual group substitution (mgs). ## qualitative assessment next, we perform qualitative evaluation to assess whether realistic counterfactuals are created. ## performance evaluation we are interested in understanding the counterfactual fairness of pre-trained models for dutch hate speech detection and evaluating methods for improving it. to this end, we focus on the bertje model, a monolingual pre-trained model used in previous research in dutch hate speech detection @xcite . this model categorizes hate speech into four classes: 'acceptable', 'inappropriate', 'offensive', and 'violent', corresponding to the categories in the imsypp dataset. the bertje model was finetuned specifically for the imsypp dataset (de vries et al., 2019), making it a natural choice for evaluating fairness in this context. additionally, we further fine-tune the baseline bertje model with counterfactual sentences generated using sll, mgs and llm-based methods. this results in five models: (1) baseline bertje, (2) bertje + sll, (3) bertje + mgs, (4) bertje + llmdef, and (5) bertje + llmlist, which will be evaluated on (counterfactual) fairness and performance. we tested all of these models on the test data from the imsypp dataset with stratified sampling. as shown in analyzing the performance per class label ( ## fairness assessment we also evaluate the fairness of these models using individual (counterfactual) and group fairness metrics. for fairness evaluation, we have categorized the sgts into seven groups based on nationality, skincolor, gender, sexuality, religion, age, and ideology (see note that the models are trained on four labels (appropriate, inappropriate, offensive or violent), and are being evaluated on the dutch counterfactual dataset which has toxic and non-toxic labels. because of the discrepancy between the predicted labels and the original labels, we consider the 'appropriate' and 'inappropriate' labels as non-toxic and the 'offensive' and 'violent' as toxic. counterfactual fairness: we use counterfactual token fairness (ctf) @xcite as a metric to assess the counterfactual fairness of our models. ctf aims to measure the fairness of the model outputs by assessing how much those outputs change when the inputs are altered with counterfactual examples. the ctf is quantified using the following equation: in this equation, x represents the set of original input instances, which have at least one of the sgts. each instance x in x has a corresponding counterfactual instance x ′ in the set x cf . the functions g(x) and g(x ′ ) compute the labels for the original sentence and the corresponding counterfactual sentence, respectively. the absolute difference |g(x)g(x ′ )| captures how much the model's predictions change between an original instance x and its counterfactual x ′ . a lower ctf value indicates that the model's outputs for original sentences and their counterfactual versions are more similar. a lower ctf is desired because it suggests that the model treats counterfactual variants of the same input fairly. ## limitations generating counterfactuals in dutch for hate speech detection presents several challenges. we observe that some counterfactuals produced using the sll and mgs methods do not adhere to proper dutch grammar and occasionally result in sentences that lack semantic coherence. this issue is particularly evident with the sll method, which relies on gpt-2, a model not optimally suited for the dutch language. in contrast, counterfactuals generated by large language models (llms) tend to be more grammatically accurate and semantically plausible in dutch. another limitation of our work lies in model performance. specifically, we find that finetuning the dutch-language bertje model on counterfactual datasets generated via sll and mgs methods improves fairness metrics. however, finetuning on datasets generated by llms does not yield similar improvements, especially for the toxic class. this is a counterintuitive outcome, suggesting that the counterfactual generation process may inadvertently introduce or reinforce biases, negatively impacting fairness for certain classes. one possible explanation is that unrealistic or noisy counterfactuals degrade the quality of the training data. additionally, dutch social group terms were generated manually, and there is a risk that some relevant terms may have been omitted. our prompt design approach also relies on baseline strategies; more sophisticated techniques, including ablation studies, are not explored in this work. finally, we do not analyze how the number of counterfactuals affects model performance and fairness. ## conclusions and future work this paper generates and evaluates the effect of counterfactual dutch-language datasets for hate speech detection. the research finds that counterfactual generation methods introduce notable fairness improvements while maintaining performance in the dutch context. the resulting datasets are publicly available for research 2 , being the first such datasets for the dutch language. our empirical evaluations indicate improved overall counterfactual fairness and classification performance. the counterfactual models, in general, slightly improve on f1 score, precision and recall. performance improvements may be partially due to increased training data for the counterfactual models, yet sometimes increased training data also exacerbates class imbalances. however, performing ablation studies with varying size of training data and finetuning strategies can be done in the future to study the impact of data variations on performance and fairness. in addition, extending this work with stateof-the-art llms can provide different results and insights. category values nationality turk, marokkaan, mocro, nederlander, hollander, indo, suri, surinamer, antiliaan, syrier, chinees, duitser, belg, afrikaan, aziaat, amerikaan, arabier, latina, latino, turks, marokkaans, mocro, nederlands, hollands, indonesisch, indo, suri, surinaams, antilliaans, syrisch, chinees, duits, belgisch, afrikaans, engels, aziatisch, amerikaanse, arabisch, latina, latino, immigrant, migrant, asielzoeker, vluchteling, autochtoon, allochtoon skincolor wit, blank, zwart, donker gender vrouw, man, transgender, nonbinair, vrouwelijk, mannelijk, masculien, intersekse sexuality homo, hetero, heteroseksueel, homoseksueel, trans, bi, lesbisch, biseksueel religion christelijk, katholiek, moslim, protestant, jood, joods age oudere, jongere, volwassene, student, oud, jong, volwassen ideology marxist, populist, feminist, liberaal, boer, communist, rechts, links, marxistisch, populistisch, feministisch, liberaal, communistisch