Evaluating Gender Bias in D utch NLP : Insights from R ob BERT -2023 and the HONEST Framework (2025)

Abstract: This study investigates gender bias in the Dutch RobBERT-2023 language model using an adapted version of the HONEST framework, which assesses harmful sentence completions. By translating and expanding HONEST templates to include non-binary and gender-neutral language, we systematically evaluate whether RobBERT-2023 exhibits biased or harmful outputs across gender identities. Our findings reveal that while the model’s overall bias score is relatively low, non-binary identities are disproportionately affected by derogatory language.

## introduction large language models are increasingly being used in a wide range of natural language processing (nlp) tasks, from chatbots to text generation. however, studies have revealed the concerning potential for these powerful models to perpetuate and amplify societal biases, including gender biases (i.a. @xcite . @xcite observed that these biases can also manifest in text generation, leading to the risk of producing sentences that are hurtful and steeped in gender stereotypes. while a growing body of research has examined gender bias in nlp models, the experiences and perspectives of transgender and non-binary individuals have often been overlooked @xcite . moreover, the majority of studies in this domain have concentrated on english language models @xcite . to address this gap, we investigate whether the stateof-the-art dutch robbert-2023 language model @xcite exhibits biases or generates harmful language when completing templates related to binary, non-binary, and transgender identities. this study builds upon the template-and lexicon-based methodology from @xcite )'s work. the authors introduce the honest measure to assess harmful biases in language models. we adapted the honest measure for dutch embeddings while ensuring the inclusion of gender non-conforming identities. ## methodology the original honest dataset by @xcite includes datasets in several languages, but not in dutch. moreover, the dataset is primarily focused on assessing binary gender bias. therefore, the second iteration of the honest dataset by @xcite attempts to incorporate a broader range of lgbtqia+ identity terms. while this updated dataset is only available in english, it encompasses both gender identities and sexual and romantic orientations. however, sexual orientation biases exceed the scope of the current study. to create the dataset, we carefully translated the english templates into dutch. the sentences were adapted where necessary to ensure they align with common dutch expressions and phrasing. additionally, the templates were made to be more inclusive of gender non-conforming identities. we crafted new templates incorporating suitable gender identity terms and gender-neutral or nonbinary language. drawing from various sources, we engaged in a thorough process to select the most appropriate and widely accepted terminology. we consulted the wikipedia page on gender-neutral language in dutch. additionally, we turned to the çavaria word list, offered by the flemish lgbti+ organization çavaria. we also examined online queer publications, such as expreszo, to gain insights from the lived experiences of non-binary and transgender individuals. we proceeded to design a comprehensive set of templates that would enable a thorough evaluation of the model's treatment of various gender identities. the templates were carefully constructed to include masked words at the end of each sentence, allowing the model to fill in the blanks. the primary objective was to observe the model's completion of these sentences and determine whether it exhibited any specific biases or generated potentially harmful language. the robbert-2023 model represents the most recent and advanced iteration of the robbert series of dutch language models @xcite . its superior performance over other dutch models makes it well-suited for the in-depth and nuanced analysis required to assess how effectively it represents and handles gender non-conforming identities. ## results the honest score of 0.0434 indicates that approximately 4.34% of the words generated by the robbert-2023 model for gender identity templates were classified as hurtful according to the dutch hurtlex @xcite . while this score is relatively low, it still indicates the presence of harmful language in the completions. to gain a more in-depth understanding of the distribution of hurtful language across different categories, we used plotly to visualize the percentage of harmful words present in the model's completions. the resulting radar plot in figure 1 shows that derogatory words have the highest percentages. these are words with offensive and negative value, or slurs (cf. @xcite . for non-binary completions, the average percentage of derogatory terms is 4.17%, which is twice as high as for other gender identities (2.58% on average). cisgender male and female identities appear equally affected by negative stereotypes related to professions and occupations (1.7%), while non-binary and trans male identities are less impacted. the trans non-binary gender identity does not seem to be targeted by harmful language. however, the presence of harmful language for other non-binary and transgender identities suggests the model may struggle to correctly process them, which presents its own challenges.