Wanted: Personalised Bias Warnings for Gender Bias in Language Models (2025)

Abstract: The widespread use of language models, especially Large Language Models, paired with their inherent biases can propagate and amplify societal inequalities. While research has extensively explored methods for bias mitigation and measurement, limited attention has been paid to how such biases are communicated to users, which instead can have a positive impact on increasing user trust and understanding of these models. Our study addresses this gap by investigating user preferences for gender bias mitigation, measurement and communication in language models. To this end, we conducted a user study targeting female AI practitioners with eighteen female and one male participant. Our findings reveal that user preferences for bias mitigation and measurement show strong consensus, whereas they vary widely for bias communication, underscoring the importance of tailoring warnings to individual needs.Building on these findings, we propose a framework for user-centred bias reporting, which leverages runtime monitoring techniques to assess and visualise bias in real time and in a customizable fashion.

## introduction many practitioners use large language models (llms) in everyday applications, like conversational agents, due to their accessibility. they are primarily hosted in large infrastructures such as hugging face 1 and require a few lines of code. however, their wide adoption comes with some limitations and risks which might be overlooked or not entirely understood by practitioners @xcite @xcite . in this context, socio-demographic bias in language models is a well-known issue which has gained much attention following the paradigm shift 1 https://huggingface.co/ in the development of language models from a performance-based to a transparency-based perspective @xcite . in particular, gender bias is the most investigated type of sociodemographic bias @xcite . most of the research in natural language processing (nlp) focuses either on bias mitigation or bias detection @xcite . the former has proposed several techniques to de-bias language models (e.g., @xcite ; utama et al. ( 2020 )). the latter instead has led to the development of many resources like datasets and tests to analyse whether and to what extent language models are biased (e.g., @xcite ; @xcite ). practitioners can use these resources to understand the limitations and risks behind llms, which should ideally guide their decision when choosing an llm to adopt. however, the current literature lacks a user-centred approach to bias in language models. while few studies have suggested frameworks to publicly inform practitioners about the presence of bias within a language model @xcite or assess the actionability of a certain bias measure @xcite , the user perspective around bias in nlp is often neglected. this is a central aspect to consider when developing resources to either detect or mitigate bias in language models, as it can increase not only the practitioners' understanding of language models' limitations but also their trust in these models @xcite . therefore, in this work, we seek to understand practitioners' perspectives regarding (i) bias mitigation (i.e., when to intervene to reduce bias), (ii) bias measurement (i.e., which metrics to use to measure bias), and (iii) bias warnings (i.e., how to inform about the presence of bias) in the context of language models. contributions. our contribution is twofold. (1) we conduct a user study targeting female practition-ers during a workshop promoting gender-inclusive ai systems to collect their perspectives on sociodemographic biases in language models, focusing especially on gender bias. (2) we propose a customizable framework to monitor bias in language models grounded on the findings of our study. ## bias statement we focus on socio-demographic biases, particularly gender bias, where we consider system behaviours to be biased when they systematically produce skewed or unfair results like, for instance, reproducing or amplifying harmful stereotypes, erasing marginalised identities, or unequally treating female and male groups. these behaviours are harmful because they can reinforce existing social inequalities, especially if we consider the widespread adoption of language models by practitioners across many domains. in section 6, we discuss an example in the financial sector but similar implications can hold in other sectors as well. ## user study in june 2024, we conducted a pilot study at an acm womencourage foot_0 workshop that aimed to promote gender-inclusive ai systems by fostering interdisciplinary dialogue and ethical reflection. acm womencourage is an event that celebrates the contributions of women in computing and supports professionals at different stages of their careers. in 2024, the theme of the event was responsible computing for gender equality, highlighting the gender gap in technology and advocating for computing tools for social progress. our workshop was structured to address the critical intersection of gender bias and language models. through a combination of theoretical presentations, handson activities, and discussions, participants were introduced to how to identify, measure, and mitigate gender bias in language models. specifically, the workshop presentation was split into two parts: bias mitigation (section 4.1) and bias measurement (section 4.2), followed by the pilot study (section 4.3). ## findings in the following sections, we discuss the main findings of our pilot study, grouped by question. ## bias warning framework as discussed in section 5.3, there are some differing opinions on how bias warnings should be reported, but the consensus tends to favour visualisations and benchmark scores. one way to produce benchmark scores and visualisations for each model prediction's bias is to monitor the model producing the output. we propose a bias warning framework that leverages ideas from deep neural network monitoring. ## conclusion the widespread adoption of language models paired with their socio-demographic biases can perpetuate societal inequalities across many use cases. while substantial efforts in nlp research have been made to measure and mitigate these biases, this re-search highlights the often-overlooked aspect of how such biases are communicated to practitioners, which instead is a crucial aspect as it can increase user trust and understanding of these models. in this paper, we address this gap by conducting a user study on bias mitigation, measurement and warning in language models, targeting female ai practitioners during a workshop promoting gender-inclusive ai systems. specifically, we focus on gender bias and further study how practitioners' choices generalise to other socio-demographic biases. our study reveals that user preferences for bias mitigation and measurement show strong consensus, in contrast to the wide variation in user preferences for bias communication, emphasising the need for tailored approaches of bias warnings. based on these findings, we develop a user-centred framework for personalised bias reporting integrating runtime monitoring techniques into language models to assess and visualise biases dynamically. future work can expand on this preliminary framework in several directions to explore its applicability and impact more broadly. for instance, researchers could evaluate the framework using established datasets from ai ethics research, such as those in the financial domain @xcite , to better understand how well it supports practitioner workflows. another promising direction is to conduct a beforeand-after user study to assess the framework's potential in fostering user trust in ai systems, following methodologies similar to di bonaventura et al. @xcite . overall, this study opens up a range of possibilities for tailoring bias communication strategies and integrating user-centred tools into real-world model deployments.