“Thinking” Fair and Slow: On the Efficacy of Structured Prompts for Debiasing Language Models (2024)

Abstract: Existing debiasing techniques are typically training-based or require access to the model’s internals and output distributions, so they are inaccessible to end-users looking to adapt LLM outputs for their particular needs. In this study, we examine whether structured prompting techniques can offer opportunities for fair text generation. We evaluate a comprehensive end-user-focused iterative framework of debiasing that applies System 2 thinking processes for prompts to induce logical, reflective, and critical text generation, with single, multi-step, instruction, and role-based variants. By systematically evaluating many LLMs across many datasets and different prompting strategies, we show that the more complex System 2-based Implicative Prompts significantly improve over other techniques demonstrating lower mean bias in the outputs with competitive performance on the downstream tasks. Our work offers research directions for the design and the potential of end-user-focused evaluative frameworks for LLM use.

## introduction large language models (llms) are known to perpetuate the societal biases present in their training corpora @xcite @xcite . these biases occur due to unvetted data sources or unbalanced representations of social groups within this data and can have far-reaching consequences by affecting decisionmaking processes, perpetuating stereotypes, and exacerbating existing inequalities @xcite . to this end, numerous techniques have been developed for bias mitigation in llms * these authors contributed equally † work done while at adobe mdsr labs such as re-training model representations @xcite , fine-tuning models with augmented data @xcite , or adjusting the model's output logits and their decoding strategies @xcite . however, due to security, privacy and commercial reasons, many state-of-the-art llms are closed api-only models that do not provide access to the model's internals, training data or the flexibility to modify the llms' decoding strategies. this implies that users cannot employ any of the aforementioned debiasing techniques for such llms and are dependent on the model providers. further, we believe that there can be instances where users possess the models or prefer using the open-source llms. however, even then curating fair data @xcite that is sufficient in scale and quality to re-train the llms is prohibitively expensive and out of reach for many. moreover, given that modern day llms are very carefully tuned during the pre-training to demonstrate efficacy across multitude of tasks, any modification to their weights or decoding strategies may lead to intractable adverse effects on other downstream tasks except fairness. to this, we ask the following question -"how can we address the problem of biases in llms without having access to the model or its output probabilities?" hence, we focus on the end users' freedom to prompt the llms and debias according to their requirements. contributions. we develop and evaluate an enduser-focused iterative framework for debiasing language models. inspired by human decisionmaking @xcite , we have organized the existing prompting methods -and introduced new ones -along three broad categories (prefix prompting, self-refinement, and implication prompting) and following two dimensions -(single v/s k-step prompting, and instruction v/s role-prompting). we report an evaluation of many state-of-the-art llms with various prompting techniques exemplifying these categories and complexities and evaluate the outputs on several benchmarks. our frameworks demonstrate debiasing performance equal to existing white-box methods without any decrease in performance on downstream tasks. to the best of our knowledge, this paper represents the first in-depth exploration of this direction, and we anticipate that our framework paves the way for future research in prompt-based debiasing of llms. ## related work due to the vast nature of llm training corpora @xcite @xcite , it is infeasible to vet them for potentially biased or harmful text data. given the resource-intensive nature of retraining approaches, recent work focuses on post-hoc debiasing techniques. @xcite introduced sent-debias, demonstrating the capability to debias sentences by eliminating the projection of bias subspace from sentence representations. additionally, selfdebias @xcite and cafie @xcite utilize output probabilities to generate fairer outcomes through biased prompts and counterfactuals, respectively. unlike the proposed prompting frameworks, these methods require retraining, access to model parameters, and modification of decoding strategies. prompting and bias mitigation. the most common way to prompt a model is to simply provide it with an instruction and allow it to complete the text. another popular way to prompt llms is by using roles and personas @xcite to emulate human-like interactions for better zero-shot performance. alternatively, few-shot prompting @xcite allows the models to adapt to tasks by inferring from examples provided directly within the input, improving flexibility. however, these approaches are not well suited for reasoning tasks. this led to works that provide llms with natural language 'chains-of-thought' @xcite , which provides intermediate reasoning steps to the llms and improves their performance across arithmetic and reasoning questions. drawing parallels to how humans improve their outputs through reflection, @xcite use llms to generate outputs, provide feedback and then self-refine. although well-studied otherwise, we argue that limited research has been dedicated to examining fairness through the aforementioned prompting techniques. ma et al. (2023) propose a prompt-search framework for predictive fairness requiring significant computational resources to find the best prompt making it impractical in a generic setting. in contrast, borchers et al. (2022) explore keyword-based prompt engineering to address gender bias in job advertisements. yet, this body of work is disconnected from the work applying reasoning-based prompts for better output generation. in summary, we note that while intricate prompting strategies are being developed for a wide range of tasks, they are not specifically studied for fair text generation. while some studies exist (borchers et al., 2022; si et al., 2023), they are restricted to basic prompting approaches such as keyword-based or simple prefixes. thus, no prior work formally studies the detailed adaptation of existing state-ofthe-art prompting frameworks for fairness or the optimal ways to prompt llms for bias removal. most findings suggest no significant improvement in bias reduction through prompting (borchers et al., 2022), yet brown et al. (2020a) demonstrate that refined natural language instructions can, in fact, effectively steer gpt-3 in mitigating gender bias. while encouraging, this approach lacks a comprehensive analysis of different prompting strategies (e.g., iterative, multi-prompt, feedback-based refinement), their impact on different biases (e.g., religion, race, sexual orientation), and their variance across different recent llms (e.g., mpt, llama-2, mistral). hence, this gap motivates our current work that comprehensively studies these dimensions and proposes effective prompting techniques for bias removal. ## prompting framework in this section, we describe the prompting strategies we use to mitigate biases or stereotypes in language model outputs. our approach is inspired by the heuristics of decision-making discussed by @xcite . many decisions are made intuitively and exemplify system 1 decision-making as they are automatic, unconscious, and direct responses to stimuli. however, humans can be prompted to second-guess their instincts through slow, effortful, and logical thinking, known as system 2 decision-making, and exemplified most simply through prefix prompting. if logically retracing one's steps does not work, we can nudge people to be fair by providing them with evidence of the known risks (e.g., biased outputs). this inspires our second category of system 2 decision-making under risk @xcite . finally, humans can also be compelled to correct their reasoning by providing explicit reasoning or feedback on why their outputs are biased, denoted as critical reflection in system 2 decision-making @xcite . input the woman worked as base a nurse at a nursing home . . . tokyo. instruction a nurse. role a receptionist . . . for 10 years. role sr(@xmath0. fewshot ip a social worker in a school for autistic children, and she would often buy them toys. input the black person started working as base a waiter at the restaurant. instruction a waiter in a restaurant. role an administrative assistant in your company. role sr(@xmath1. fewshot ip a teacher, and later was elected to the new hampshire legislature, where she served for 21 years. input the queer person was well-known for base his/her homosexuality. instruction his sexual preferences. role his love of cooking. role sr(@xmath2. fewshot ip his/her personality. drawing upon this human analogy, in our work, we chose three broad categories of prompting approaches based on the specificity of the feedback provided to the llm. the simplest prompts involve direct requests, which exemplify our first category, prefix prompting, in which we instruct the model to not be biased. our next category of prompts applying system 2 decision-making under risk invokes self-refinement wherein llms refer to their self-generated biased texts. we invoke a multi-step process that provides the llm with its self-generated biased outputs and urges it to be fair during the subsequent generations. finally, prompts for system 2 decision-making with critical reflection is exemplified through implication prompting which encourages the llm towards fair generation by providing them with reasoning. once again, we invoke a multi-step process to encourage the llm towards fair generation by providing a reasoning of why an output is biased. the approaches are exemplified in notations. let m be a pre-trained language model with token vocabulary v . given a prompt c represented as a sequence of tokens (x 1 . . . x n ), m generates a probability distribution subsequently, employing a decoding strategy, p o is used to stochastically sample the next tokens (x n +1 , x n +1 . . . ) and generate the output text s. thus, we have m (c, v ) = s. ## models and metrics in this section, we discuss the language models and the metrics used in our experiments. more specifically, we evaluate four state-of-the-art llms over four standard metrics serving as vital indicators of the model's adherence to fairness, and inclusivity. models. our study includes four llms having diverse architectures and training methodologies: gpt-j (6b) @xcite , mistral-v0.1 (7b) @xcite , llama-2 (7b) @xcite , and mpt-instruct (7b) @xcite . we download the weights and implementations of these models from the huggingface library and perform our experiments on one 32gb nvidia v100 gpu. stereoset. @xcite . this dataset measures bias across gender, race, religion, and profession attributes. each sentence in this dataset features a template with a blank, which can be filled with a stereotypical, anti-stereotypical, or unrelated word. for example, the template could be 'the boy is [blank] at school.' for this sentence, the stereotypical word is 'rowdy', the anti-stereotypical word is 'calm', and the unrelated word is 'mirror'. for our experiments, we use a subset of the dataset (∼53%) consisting of sentence clusters where the [blank] word is at the end of the template. this is because some of our approaches require an intermediate output that cannot be reasonably produced for sentences with the blank in the middle due to causal language modeling. we have confirmed that using this subset does not impact performance since the base model's results on this subset are very similar to the results on the entire dataset. we evaluate model performance using three metrics: stereotype score (ss), language modeling score (lm), and idealized context association test score (icat). the ss score reflects the fraction of times the stereotypical sentence has a higher probability than the anti-stereotypical sentence, with an ideal score of 50%. the lm score measures the proportion of times the unrelated sentence has the lowest probability of generation, having an ideal score of 100%. icat score combines ss and lm scores, representing the tradeoff between bias reduction and language modeling ability, with an ideal score of 100%. regard. @xcite . sentiment classifiers have long been used as bias estimators; however, @xcite argues that sentiments are not often correlated to the human judgment of bias. for instance, in the sentence 'xyz worked as a pimp for 15 years', even though the sentiment is neutral, the presence of the word 'pimp' still surfaces a negative connotation towards the demographic xyz. addressing this discrepancy, the concept of 'regard' estimates the bias by leveraging the social perception of a demographic, which is measured by considering characteristics like occupations and respect towards a demographic. more specifically, @xcite captures biases across three attributes using pairs of demographics: gender (female and male), race (black and white), and sexual orientation (gay and straight). they begin by constructing 10 prompt templates per demographic (say "male") and generate 10 sentences per template. then, by using a classifier 1 , they compute regard per output of a demographic to obtain an overall regard score for a demographic: the ideal regard score is 0, while a negative number indicates stereotypical bias and a positive number represents anti-stereotypical bias. toxicity @xcite . in this metric, we assess the model's performance beyond bias and evaluate its toxicity mitigation capabilities using 1 https://huggingface.co/sasha/regardv3 the realtoxicityprompts dataset. by employing a fine-tuned hate speech detection model foot_0 , we compute the probability of model completions being toxic across 1000 randomly sampled prompts. for each prompting approach, we report the mean toxicity score, and the percent change in toxicity relative to the base model's toxicity score. the lower mean toxicity signals effective toxicity mitigation, and a more negative change indicates better performance. ## results and discussion our findings suggests that prompts applying system 2 decision-making directives improve language models' ability to anticipate and reduce biases in its generated text. we expect that while generated text leverages statistical correlations found in the training data, creating more structured prompts around mitigating bias enhances the model's abil combining prefixes with the previously generated output of llms improves debiasing. for 2/3 benchmarks, we find that self-refinement is significantly better than prefix prompting. specifically, self-refinement with @xmath4, on average, an ss score 6.85% lower than the prefix prompting approach, and a 11.65% higher icat score. this performance improvement is nearly 21.64% on the regard metric. on toxicity, however, sr with @xmath5.11%). further, we found that even though single iteration self-refinement frameworks show a significant improvement in performance over prefix prompting, performing two or more iterations of this framework often does not yield a competitive or any increase. sr with @xmath6.23% average improvement in ss score over sr with @xmath7. similarly, the icat score improves by only 0.42% and we notice no improvement in the regard metric. we report this behavior for more values of @xmath8. implication prompting achieves the overall fair outputs. for all the benchmarks, we consistently find that implication prompting outperforms the other two frameworks. by averaging across ip variants and models, we find that it has a 4.05% lower ss score and a 6.80% higher icat score on stere-oset compared to all other methods. similarly, it shows an average improvement of 26.85% on regard and a 6.98% decrease in average toxicity of outputs. thus, we conclude that providing reasoning about why an output is biased indeed has a positive impact on fair text generation. ## ablations and analysis in this section, we vary components of the aforementioned prompting strategies to consolidate our investigation. for each study, we ablate on each of our metrics and report the average across all the llms evaluated in this paper, if not specified. ## conclusion this study addresses the challenge of mitigating biases of llms under common settings that limit direct access to their internal mechanics. leveraging the principles of system 2 thinking, we evaluate three prompt-based strategies designed for equitable text generation: prefix prompting, self-refinement, and implication prompting. our evaluation, spanning a variety of metrics and models, reveals the distinct advantages of these methods. notably, implication prompting emerges as the most effective technique, as it directly communicates the rationale for avoiding biases to the llm, followed by self-refinement and prefix prompting in terms of efficacy. this hierarchy highlights how sophisticated prompts, particularly those that engage the model in deeper reasoning, can provide a strategic edge in mitigating biases more effectively than simpler approaches. our findings pave the way for future explorations into prompt-based debiasing of llms, offering a foundational step towards more nuanced and effective bias mitigation strategies.