Dual Debiasing: Remove Stereotypes and Keep Factual Gender for Fair Language Modeling and Translation (2025)

Abstract: Mitigation of biases, such as language models’ reliance on gender stereotypes, is a crucial endeavor required for the creation of reliable and useful language technology. The crucial aspect of debiasing is to ensure that the models preserve their versatile capabilities, including their ability to solve language tasks and equitably represent various genders. To address these issues, we introduce Dual Dabiasing Algorithm through Model Adaptation (2DAMA). Novel Dual Debiasing enables robust reduction of stereotypical bias while preserving desired factual gender information encoded by language models. We show that 2DAMA effectively reduces gender bias in language models for English and is one of the first approaches facilitating the mitigation of their stereotypical tendencies in translation. The proposed method’s key advantage is the preservation of factual gender cues, which are useful in a wide range of natural language processing tasks.

## introduction gender representation in large language models (llms) has been the topic of significant research effort @xcite . past studies have predominantly focused on such representation to identify and mitigate social biases. admittedly, biases are a challenging issue limiting the reliability of llms in real-world applications. however, we argue that preserving particular types of gender representation is crucial for fairness and knowledge acquisition in language models. to provide a more detailed perspective, we draw examples of both unwanted and beneficial types of gender signals in llms. undesirable biases are typically inherited from stereotypes and imbalances in the training corpora and tend to be amplified further during model training @xcite ., * work done at charles university the hairdresser marie curie she the nurse die ärztin he the scientist der arzt they the doctor the doctor "die ärztin" and "der arzt" are respectively female and male german translation for "the doctor". to address these intricate ways gender signals are present in natural language, we introduce a new method, 2dama, that post-hoc modifies pretrained language models to represent gender in an equitable way, i.e., without stereotypical bias but with factual gender information. as the core contribution, we introduce the novel method of dual debiasing that aims at our core problem of decreasing bias while maintaining an equitable representation of the factual gender. specifically, we aim to reduce the models' reliance on stereotypes in predictions, e.g., given a stereotypical prompt as the one in figure 2 : "the salesperson laughed because", we intend to coerce equitable probabilities of possible gender predictions manifested by pronouns "he", "she", or "them". on the contrary, when considering a prompt that contains factual gender information: "the king laughed because" the desired output distribution would assign a high probability to the male pronoun. ## methodology in this section, we formally introduce dual debaising algorithm through model adaptation (2dama), a new dual debiasing method, and provide theoretical backing for the presented approach. appendix a contains the proofs and further terminological explanations. ## experimental setting this section presents an empirical setting to examine the practical application of model editing methods. we describe models, data, and evaluation metrics for gender bias and general performance. ## debiasing language models in the first batch of the experiments, we evaluate the effectiveness of debiasing language models. in these experiments, we solely focus on tasks in english. we specifically analyze three model editing approaches: dama as a baseline; dama in combination with leace; and 2dama, which employs dual debiasing to preserve factual gender information. ## beyond english: multilingual debiasing in a multilingual setting, we debias a model finetuned for translation: alma-r 13b @xcite by employing the collection of the new multilingual debiasing prompts. we specifically evaluate gender bias and quality of translation between english and czech, german, and russian. foot_1 ## conclusion we highlight the importance of considering the dual character of gender encoding in model editing. the theoretical and empirical results show that our novel model editing methods: 2dama effectively reduces the impact of stereotypical bias on the predictions while preserving equitable representation of (factual) gender based on grammar and semantics. maintaining the factual component of gender representation is crucial for debiasing in lan-guages other than english, for which gender markings are ubiquitous. furthermore, our method does not significantly deteriorate the high performance of llms in various evaluation settings unrelated to gender.