Open- D e B ias: Toward Mitigating Open-Set Bias in Language Models (2025)

Abstract: Large Language Models (LLMs) have achieved remarkable success on question answering (QA) tasks, yet they often encode harmful biases that compromise fairness and trustworthiness. Most existing bias mitigation approaches are restricted to predefined categories, limiting their ability to address novel or context-specific emergent biases. To bridge this gap, we tackle the novel problem of open-set bias detection and mitigation in text-based QA. We introduce _OpenBiasBench_, a comprehensive benchmark designed to evaluate biases across a wide range of categories and subgroups, encompassing both known and previously unseen biases. Additionally, we propose _Open-DeBias_, a novel, data-efficient, and parameter-efficient debiasing method that leverages adapter modules to mitigate existing social and stereotypical biases while generalizing to unseen ones. Compared to the state-of-the-art BMBI method, Open-DeBias improves QA accuracy on BBQ dataset by nearly **48%** on ambiguous subsets and **6%** on disambiguated ones, using adapters fine-tuned on just a small fraction of the training data. Remarkably, the same adapters, in a zero-shot transfer to Korean BBQ, achieve **84% accuracy**, demonstrating robust language-agnostic generalization. Through extensive evaluation, we also validate the effectiveness of Open-DeBias across a broad range of NLP tasks, including StereoSet and CrowS-Pairs, highlighting its robustness, multilingual strength, and suitability for general-purpose, open-domain bias mitigation. The project page is available at: [https://sites.google.com/view/open-debias25](https://sites.google.com/view/open-debias25)

## introduction the advent of large language models (llms) has transformed the field of natural language processing (nlp), enabling breakthroughs in diverse tasks such as machine translation @xcite , summarization @xcite , question answering (qa) @xcite etc. these llms, with their superior ability to understand and generate human-like text, have become integral to modern ai applications. however, alongside their remarkable capabilities, llms often inherit and amplify the biases present in their massive training corpora @xcite , which can manifest in downstream tasks like qa @xcite , leading to unfair, inaccurate, or even harmful responses. this duality-unprecedented utility coupled with inherent bias-poses a critical challenge for llms deployment in real-world scenarios. we define bias as systematic, unbalanced associations learned by language models that reflect or reinforce stereotypes, preferences, or assumptions. while this includes social biases such as gender, race, or age, it also extends to subtler forms like geographic preferences, brand favoritism, occupational associations, and aesthetic judgments. these are not simple frequency artifacts, but persistent patterns that distort meaning, reinforce skewed associations, and subtly shape model behavior. for example, a model may associate poverty with rural regions, professionalism with specific global brand, or beauty with western minimalist design. such patterns can accumulate and influence downstream decisions over time. motivation: existing efforts to address bias in nlp systems, such as those leveraging benchmarks like bbq @xcite , have largely focused on closed-set bias detection that operates on a predefined and limited group of well-known bias categories, such as gender or race, that are established in advance. it restricts analysis to only these known concepts, thereby preventing the discovery of novel or unexpected biases. for example, in response to the question "which workspace looks more professional?", a model may systematically prefer minimalist white offices, reflecting an implicit western aesthetic bias as a proxy for professionalism. as illustrated in figure 1 , a closedset method might detect associations between the prompt "a person using a laptop" and demographic traits like gender or race, but fail to capture subtler biases related to laptop brand, workspace aesthetics, or socio-economic signaling. while such approaches are valuable for identifying and mitigating well-known forms of bias, they fall short in capturing open-set biases that exist in the prompt but lie outside predefined categories, as evidenced by stable diffusion's propagation of novel biases in laptop brands and professional stereotypes through open-set analysis @xcite . hence, there is an urgent need for open-set bias mitigation in qa systems. approach: as open-set bias mitigation remains an unexplored problem, no existing benchmark supports systematic evaluation of emergent biases beyond fixed social categories. we first address this gap by curating a dedicated dataset (named openbiasbench) tailored for open-set bias analysis. inspired by d' @xcite , we leverage gemini-1.5-flash @xcite to build a knowledge base of potential biases. by prompting gemini with a collection of target textual captions from ms coco @xcite , we uncover specific biases associated with various entities in the captions. this methodology allows us to discover both known and novel biases, potentially embedded within the llm. the existing debiasing methods are limited to fixed bias categories, and cannot handle open-set scenarios. to overcome this limitation, we also propose a novel debiasing framework tailored for openset bias mitigation in qa tasks. our method employs lightweight adapters for parameter-efficient fine-tuning of pre-trained language models (plms) and is trained only on a small subset of representative bias categories. these adapters effectively mitigate existing biases while aiding generalization to unseen and emergent forms of biases. our adapter-based debiasing module allows easy integration with most language models. we rigorously evaluate the performance of our debiased model on the challenging openbiasbench dataset. to realize the true potential of the method, we also analyze its efficacy in debiasing other tasks beyond qa such as natural language inference, single-sentence classification, paraphrase detection, semantic similarity regression and open-ended sentence ranking. below, we summarize our key contributions: ## related work bias in nlp models has become a critical concern as these systems are increasingly deployed in real-world applications. efforts to understand and mitigate such biases have produced a variety of benchmarks, techniques, and frameworks, most of which operate under a closed-set assumption. transformer-based models, such as bert and gpt, have been at the forefront of nlp advancements, but are also prone to encode societal biases @xcite . several benchmarking datasets have been introduced to measure bias in nlp models. for example, datasets like weat @xcite evaluate biases in word embeddings through association tests, while others focus on task-specific benchmarks for sentiment analysis @xcite , toxicity detection @xcite , and qa. bbq is a notable dataset for qa bias evaluation, designed to assess stereotypical associations across various social dimensions @xcite . these benchmarks provide diverse metrics for quantifying bias but are often limited to predefined categories, restricting their applicability to open-set scenarios. mitigation strategies for nlp models include debiasing word embeddings @xcite , counterfactual data augmentation @xcite , fair representation learning @xcite , and algorithmic fairness @xcite constraints. recent efforts have also explored adapter-based approaches for debiasing. sustainable modular debiasing @xcite introduces an efficient, modular technique that employs lightweight adapter modules to isolate bias information and allow for flexible, composable debiasing across tasks. adapterfusion @xcite extends this idea by dynamically combining multiple task-specific adapters for transfer learning without catastrophic forgetting. these methods offer promising avenues for scalable debiasing, yet they remain largely confined to closed-set settings where the biases are known and well-defined during training. qa systems are particularly susceptible to biases due to their reliance on contextual information @xcite . existing works on qa bias mitigation focus on closed-set scenarios using datasets like bbq @xcite or adversarial training methods. while effective for known biases, these methods struggle with unseen categories. open-set approaches like @xcite have emerged recently in other domains (e.g., text-to-image generation), leveraging generative models to identify novel biases without predefined categories. to address this gap, we introduce an open-set qa dataset along with adapter based debiasing framework, enabling effective debiasing even for unseen bias categories not present during training. ## openbiasbench dataset open-debias focuses on mitigating biases in an open-set setting, aiming to uncover and address emerging and context-sensitive biases rather than only correcting a predefined set of social biases. for evaluating our method, an open-set bias dataset is essential, one that includes a wide spectrum of bias types beyond traditional social categories. for example, biases related to colors, geographic locations, professions, or object attributes, which are often overlooked and not systematically covered by existing benchmarks like bbq, and un-qover @xcite as they focus primarily on closed-set, well-known social biases. to systematically study a broader and more realistic spectrum of biases in qa systems, we construct openbi-asbench (d open ), a large-scale dataset tailored to handle open-set bias categories. now, we detail our dataset curation process, which is also summarized in algorithm 1. ## adapter based debiasing in a closed-set setting, models are trained and tested on a predefined set of categories. however, real-world applications often involve situations where a trained model encounters novel examples that do not belong to any of the known categories. this setting is referred to as the openset scenario, where the model must be capable of recognizing and appropriately handling previously unseen categories during training. in this work, we propose an adapter based debiasing module (open-debias) to mitigate bias in the open-set scenario. ## results and analysis we evaluate our method (open-debias) on deberta-v3-large (deb-l) @xcite and roberta-large (rob-l) @xcite , two state-of-the-art transformer models that have demonstrated strong performance on both question answering (qa) tasks @xcite @xcite and bias detection or mitigation @xcite . to ensure a fair and consistent comparison, we adopt encoder-based models, aligning our setup with that of the existing methods. for controlled evaluation, we use the bbq dataset and employ our openbiasbench dataset for open-domain analysis. we format the inputs using the race schema. in all the training expts, we fix λ = 0.1 (eq. 6) for ambiguous and 0 for disambiguous contexts. adapters and fusion layers are trained for five epochs, while keeping other hyperparameters similar to @xcite . note that in all the result tables, the highlighted categories indicate the ones used for adapter training. to ensure minimal computation, we keep the base model weights frozen and train only categoryspecific adapters using a few (500) examples per category (c train ). we also discuss the impact of training data size (varying examples per category) in the appendix ## conclusion we introduced open-debias, a novel adapter-based framework for open-set bias detection and mitigation in qa systems, capable of addressing both known and novel bias categories. our approach is parameter-efficient, maintains core qa capabilities, and demonstrates strong multilingual generalization. it achieved substantial improvements in handling ambiguous content, along with notable gains in disambiguated scenarios. to support open-set evaluation, we developed a dataset that broadens bias benchmarking across a wider range of socially relevant attributes, going beyond the limitations of traditional closed-set settings.