F air S teer: Inference Time Debiasing for LLM s with Dynamic Activation Steering (2025)

Abstract: Large language models (LLMs) are prone to capturing biases from training corpus, leading to potential negative social impacts. Existing prompt-based debiasing methods exhibit instability due to their sensitivity to prompt changes, while fine-tuning-based techniques incur substantial computational overhead and catastrophic forgetting. In this paper, we propose FairSteer, a novel inference-time debiasing framework without requiring customized prompt design or model retraining. Motivated by the linear representation hypothesis, our preliminary investigation demonstrates that fairness-related features can be encoded into separable directions in the hidden activation space. FairSteer operates in three steps: biased activation detection, debiasing steering vector (DSV) computation, and dynamic activation steering. Specifically, it first trains a lightweight linear classifier to detect bias signatures in activations, and then computes DSVs as intervention directions derived from small contrastive prompt pairs. Subsequently, it performs debiasing by adjusting activations with DSVs in the inference stage. Comprehensive evaluation with six LLMs demonstrates the superiority of FairSteer across question-answering, counterfactual input evaluation and open-ended text generation tasks. Code will be released.

## introduction large language models (llms) have demonstrated remarkable performance in various tasks @xcite @xcite . however, they tend to inherit social biases from their training corpus, leading to behaviors that negatively impact underrepresented groups @xcite @xcite @xcite . this challenge thus requires effective debiasing techniques to better align llms with ethical ai principles. existing debiasing techniques focus primarily on two paradigms: (1) in-context prompting-based methods @xcite @xcite leverage carefully crafted instructions to guide llms toward unbiased outputs, yet their effectiveness relies heavily on the quality of the prompt and is not robust to variations in phrasing. (2) fine-tuning-based methods @xcite @xcite @xcite typically retrain models using balanced datasets or apply methods such as projection-based techniques, component-specific debiasing, contrastive learning and reinforcement learning. they provide more direct control over biases, with the cost of high computational expense, risk of catastrophic forgetting, and dependence on large annotated datasets that are hard to collect. therefore, the question arises: can we mitigate bias in llms without retraining, instead deferring the debiasing process to the inference stage? this approach, which we categorize as inference-time debiasing, is simpler and more practical than intraining debiasing, as it avoids the need for complex training procedures or extensive computational resources. recent research on inference-time debiasing focuses on modifying decoding strategies @xcite @xcite @xcite by suppressing biased tokens. however, such strategies usually reduce the diversity of outputs and degrade the performance of llms on non-debiasing tasks. our work is initially inspired by the linear representation hypothesis @xcite @xcite @xcite : semantic features like truth @xcite , sentiment @xcite , humor @xcite , and refusal @xcite are encoded as linearly separable directions in llm activation spaces. this raises a fundamental question: can fairness-related concepts similarly be encoded as separable geometric structures in hidden states, enabling bias mitigation through activation steering? to investigate this, we perform a preliminary analysis across six llms, as shown in figure 2 . our key observation reveals that bias signatures exhibit over 90% linear separability in intermediate layers, indicating the feasibility of geometric intervention. motivated by our preliminary findings, we propose fairsteer, a novel inference-time debiasing method that dynamically adjusts activation without retraining. our approach operates in three synergistic stages, as shown in figure 1 . (1) biased activation detection (bad): we train linear classifiers @xcite on intermediate layer activations to detect bias signatures across llm layers with a predefined bias classification dataset. (2) debiasing steering vector (dsv) computation: we compute geometrically interpretable intervention directions using contrastive prompt pairs @xcite , isolating bias-specific features by controlling contextual variables. dsvs require only one hundred annotated examples, significantly fewer than the datasets for fine-tuning-based debiasing methods. we conduct comprehensive experiments over six llms (llama, vicuna, mistral) and four popular datasets (bbq @xcite , unqover @xcite , crows-pairs @xcite and ceb @xcite ), with tasks across question answering, counterfactual input evalua- ## methodology based on our preliminary findings, we introduce fairsteer which operates in three stages: 1) biased activation detection, 2) debiasing steering vector computation, and 3) dynamic activation steering during inference, as shown in figure 1 . ## related work promoting fairness in llms is a critical component of alignment, a broader objective that seeks to ensure ai systems adhere to human values @xcite @xcite @xcite . researchers have proposed various debiasing methods, which can be group into two main paradigms: (1) incontext prompt-based methods @xcite @xcite leverage carefully crafted instructions or contextual prompts to guide llms toward unbiased outputs during generation. these approaches avoid modifying model parameters and are easily applicable to black-box models. however, they are sensitive to prompt formulation, struggle with implicit biases, and depends heavily on prompt quality and specific use cases. (2) fine-tuning-based methods involve constructing rebalanced datasets and retraining models @xcite @xcite @xcite , or using techniques such as projection-based methods @xcite @xcite , debiasing specific components @xcite @xcite @xcite , contrastive learning @xcite @xcite , adversarial learning @xcite , and reinforcement learning @xcite approaches. although effective, these methods involve complex and resource-intensive training, requiring diverse debiasing datasets. in contrast, inference-time debiasing methods are simpler and more practical, as they do not require retraining or altering the model's architecture. existing inference-time techniques primarily focus on decoding strategy modification, such as constrained next-token search @xcite @xcite or generating and reranking alternative outputs @xcite @xcite @xcite . however, the key challenge with these methods is balancing effective bias mitigation while preserving output diversity @xcite . furthermore, many of these techniques @xcite are tailored to older models such as bert @xcite and gpt-2 @xcite , limiting their applicability to more recent architectures. our approach advances inference-time debiasing by demonstrating that fairness-related features are linearly separable in activation space, enabling targeted interventions via geometrically interpretable steering vectors. ## conclusion in this paper, we propose fairsteer, an inferencetime debiasing framework for llms without requiring retraining. inspired by the linear representation hypothesis, our preliminary analysis demonstrates that fairness-related features can be encoded into separable directions in the hidden activation space. fairsteer detects bias signatures in these activations using a lightweight linear classifier and applies debiasing steering vectors, which are computed by a small set of contrast prompt pairs, to dynamically adjust these activations during inference. experimental results show that fairsteer effectively mitigates bias in tasks including question-answering, counterfactual input evaluation, and open-ended text generation, showcasing its broad applicability while preserving language modeling capabilities.