Disentangling Biased Representations: A Causal Intervention Framework for Fairer NLP Models (2025)

Abstract: Natural language processing (NLP) systems often inadvertently encode and amplify social biases through entangled representations of demographic attributes and task-related attributes. To mitigate this, we propose a novel framework that combines causal analysis with practical intervention strategies. The method leverages attribute-specific prompting to isolate sensitive attributes while applying information-theoretic constraints to minimize spurious correlations. Experiments across six language models and two classification tasks demonstrate its effectiveness. We hope this work will provide the NLP community with a causal disentanglement perspective for achieving fairness in NLP systems.

## introduction since nlp models are trained on human-generated texts, they inevitably inherit and amplify social biases, leading to non-neutral representations where sensitive attributes (e.g., gender, race, or religion) spuriously correlate with task-related attributes. for instance, in hate speech detection, tweets mentioning minority groups are more likely to be falsely flagged as toxic, while sentiment analysis systems may associate certain dialects with negative polarity. such biases not only undermine model accuracy and reliability but also sustain the prevalence of allocation harms, such as unequal access to services; furthermore, they give rise to representational harms, like reinforcing stereotypes. while large language models (llms) have achieved remarkable capabilities, their widespread application has paradoxically amplified these bias issues, as their training on large-scale web data often amplifies existing social biases @xcite @xcite . most methods predominantly conceptualize biases as an issue rooted in statistical correlations. for instance, the co-occurrence of gender-biased lexical items within the training dataset has the potential to induce skewed model predictions. however, this correlation-centric paradigm falls short in discerning between spurious patterns and authentic causal relationships. a more fundamental solution emerges when we reconceptualize the social biases through causal inference. by identifying sensitive attributes such as gender as confounding variables that spuriously influence both input features and output labels, we can develop interventions that address bias at its source. this causal perspective, particularly through pearl's framework of counterfactual analysis @xcite , enables techniques like counterfactual data augmentation @xcite -where models are trained on carefully constructed "what-if" scenarios to break their reliance on sensitive attributes while preserving task-relevant features. although the counterfactual data generated is effective, it needs to have a certain degree of rationality in real-world scenarios. otherwise, it may introduce misinformation and subsequently misguide the model's learning trajectory. moreover, it is highly probable that the computational and storage expenses associated with the generation of large volumes of data will experience a substantial increase. to mitigate these limitations while retaining the benefits of counterfactual evaluation, we using counterfactual data solely for testing robustness rather than for model training. in this work, we perform a causal analysis of social biases in nlp models, identifying that the core issue stems from latent representation entanglement-where lms implicitly encode sensitive and task-related attributes through shared representational spaces. grounded in this causal perspective, we design a prompt-guided intervention framework that achieves: (1) explicit attribute separation through attribute-specific prompting strategies, where distinct prompts isolate sensitive and taskrelated features in the latent space; (2) causal disentanglement via mutual information minimization, effectively cutting the spurious correlation pathways between attributes; and (3) counterfactual robustness validation, ensuring model predictions remain invariant to sensitive attribute perturbations. ## bias statement the biases examined in this work arise when lm representations systematically encode and amplify spurious correlations @xcite between sensitive (or protected) attributes and task-related predictions. sensitive attributes refer to demographic or identity-related characteristics, such as gender, race, age, or religion that should not influence the fair predictions of lms @xcite . in the absence of mitigation for sensitive attributes may lead to some concrete harms: allocation harm occurs when model outputs misclassify or disadvantage specific demographic groups @xcite @xcite , while representational harm manifests when models perpetuate stereotypes by embedding social biases into their latent representations, exemplified by gender-occupation or race-profession associations @xcite . these biases originate from three primary sources: pretraining data that reflect historical inequalities, the model's propensity to exploit shortcuts for prediction, and the fundamental statistical nature of machine learning that conflates correlation with causation. such biases induce unfair algorithmic outcomes that adversely affect protected demographic groups and reinforcing harmful stereotypes in many ai systems. ## related work bias in nlp systems. a rising amount of research has delved into issues of bias in nlp systems. in the early stages, numerous studies @xcite @xcite focused on uncovering stereotypes within word embeddings. more recently, as llms have gained prominence, new challenges in detecting and mitigating in llm have become the focus of bias research @xcite . this becomes more challenging owing to the complex nature of llms, which are trained on a large amount of text data that may intrinsically contain diverse forms of biases. biases are widespread across different llms @xcite , and llms also exhibit more patterns of bias @xcite . causal methods for bias mitigation. causal inference provides a theoretical framework for addressing these challenges. @xcite employed causal mediation analysis to analyze the causal roles of different components within the model in the model's behavior. @xcite proposed causal-debias to unify the debiasing of pretraining and fine-tuning, reducing biases in finetuned models. building upon but distinct from previous studies, our approach leverages lm representations and strategic prompting to obtain disentangled features for bias mitigation. we are inspired by representation learning theory @xcite , particularly the identifiability theory that formalize the conditions for factor disentanglement. @xcite proposed an adversarial disentangled debiasing model to dynamically decouple social bias attributes from intermediate representations during main task training, but their framework was not situated within a causal inference paradigm. ## causal foundations and problem formulation in this section, we establish a causal framework to analyze the bias propagation in lm. we start by introducing some fundamental concepts @xcite and then propose a causal graph (figure 2 ) to characterize the entanglement of attributes in lms representations.