Prompting Fairness: Learning Prompts for Debiasing Large Language Models (2024)

Abstract: Large language models are prone to internalize social biases due to the characteristics of the data used for their self-supervised training scheme. Considering their recent emergence and wide availability to the general public, it is mandatory to identify and alleviate these biases to avoid perpetuating stereotypes towards underrepresented groups. We present a novel prompt-tuning method for reducing biases in encoder models such as BERT or RoBERTa. Unlike other methods, we only train a small set of additional reusable token embeddings that can be concatenated to any input sequence to reduce bias in the outputs. We particularize this method to gender bias by providing a set of templates used for training the prompts. Evaluations on two benchmarks show that our method is on par with the state of the art while having a limited impact on language modeling ability.

## introduction large language models (llms) have claimed stateof-the-art performance on most of the classical natural language processing (nlp) tasks in recent years while facilitating new frontiers in language generation. however, besides being computationally expensive, their performance comes at an additional cost, as they tend to pick up social biases from the vast data required for their pretraining. consequently, these models can exhibit representational harms, such as disparate system performance, exclusion or stereotyping, or allocation harms, such as discrimination and unequal allocation of resources @xcite . with an increased number of use cases and adoption rates, ensuring fairness is becoming more and more critical. our work can be summarized by the following key contributions: ## prompt tuning for bias mitigation it has been shown that concatenating prompts to the input of a pretrained language model is a viable method of altering its behaviour for different use cases. notably, in-context learning, which involves prompting with a few training examples, can be successfully used for adapting a model to various downstream tasks. in @xcite , the authors formalize in-context learning as an implicit bayesian inference, such that the probability of the model's output o can be expressed as we argue that a similar approach can be used for debiasing encoder-only llms. during pretraining, the model learns to maximize the likelihood of the training data. this behaviour might not always be desirable, especially if, due to the characteristics of the training data, maximizing its likelihood involves relying on various stereotypes. as opposed to removing or hiding information from the model, either at training or at inference time, we aim to give the model additional information at inference, in the form of compact prompt embeddings, which could enable it to implicitly infer a latent concept encompassing the desired behaviour: generating a fair and unbiased output. the prompts should be able to encompass the desired behaviour as accurately as possible. ideally, we want the model to produce output which is unbiased while also retaining the identity of all social groups and maintaining correctness in general language modeling. trying to express this in handcrafted prompts would not be straightforward, especially if the model to be unbiased was not explicitly trained to follow human instructions. instead, we base our approach on "prompt tuning" @xcite , which involves concatenating a set of trainable embeddings to the embedded input of the model while keeping the other parameters frozen. templates the prompt embeddings are trained using a dataset of templates with bias slots and target slots. each bias slot can be replaced by words specific to each social group affected by the type of bias to be mitigated. for example, for gender bias, we could have a bias slot which can be replaced by either "he" or "she", another one which could be replaced by either "his" or "her" and so on. the target slots represent the words in the templates which the model should predict. for target slots, there is a set of allowed options, composed of: the reason for explicitly defining the expected outputs of the model and dividing it into general and specific is to avoid training prompts which cause the model to "forget" the identity of each group. for simplicity, we use a single target slot per template. training and loss function we train the prompts by replacing the bias slots of each template with their specific variants for each group and minimizing the kl divergence between the probability distribution predicted by the model for the allowed options of the target slots and a reference probability distribution. for a given template t and social group a, the bias slots in t are replaced with corresponding substitutions for a to obtain t a . we denote by the set of group-specific options for group a. then, denoting by t the target slot for t a , we obtain the probability distribution p t a defined on the sample space: here p t a (@xmath0. to obtain a proper probability distribution, we also consider the probability of t not being in the set of allowed options. t a for t a as the average probabilities predicted by the original model (denoted by p i ) across the set g g g of all social groups considered: we define the loss term for template instantiation t a , obtained by filling bias slots in template t with terms specific for social group g, as the kl divergence between the probability distribution predicted by the model and the reference probability distribution: the reference probability distribution p * t a for each template instantiation t a is treated as a constant and can be precomputed beforehand. then, the total loss for a set t t t of templates is obtained by instantiating each template t with the bias slot terms for each social group g, and summing over the loss terms for each resulting template instantiation t g : in previous formulas, we assumed for simplicity a single template instantiation t g for each pair of a template t and a group g. in the general case, there may exist multiple such template instantiations t k g , depending on whether some bias slots in t can be filled by multiple pairs of values. in this case, we sum over all considered 2 instantiations: gender debiasing bert and roberta we particularize this method for reducing gender bias in bert @xcite and roberta @xcite models. we constructed a dataset of 159 templates, mostly focused on genders in relation to professions/occupations, as this is one area in which we empirically observed the models to generate biased predictions. examples of templates are listed in for simplicity, we restrict the choice of allowed options to words that each model's tokenizer can represent with a single token and subsequently replace the target slots in the templates by a single [mask] token. this is not a big limitation in this case, since bert's and roberta's vocabularies can represent most common english words by a single token. however, it might pose problems for models with other types of tokenizers and for different languages, as it would require either using a limited number of options or creating separate 2 for practical reasons, we might consider only a subset of all possible instantiations templates for different numbers of mask tokens. we selected the allowed options empirically by hand-picking appropriate completions and choosing from the original model's predictions for the templates. while templates are focused on professions/occupations, the allowed options are not restricted only to this specific domain; for some of the templates, there are other valid completions. we selected 219 general options and 15 pairs of group specific options. some examples are listed in implementation we use pretrained models from huggingface transformers @xcite and use the prompt tuning implementation in from peft @xcite with py-torch @xcite for building and training our debiased models. training is done using an adamw @xcite optimizer with an initial learning rate of 1e -2 and a linearly decreasing schedule with warmup. since only prompt parameters are updated and the dataset used is small, training converges fairly rapidly: training of each debiased model takes about half an hour on an nvidia 1050ti gpu. ## results we evaluate our method for mitigating gender bias in bert and roberta on the gender tests from seat @xcite and stereoset @xcite . for stereoset, a stereotype score (ss) closer to 50% indicates a less biased model. in case of seat, we average the last layer's hidden representations and normalize the resulting vector, as @xcite ; @xcite , but exclude the representations corresponding to the prompt tokens from this computation. for analyzing the loss in language modeling performance, we use the language modeling (lm) score from stereoset and the pseudo-perplexity @xcite on the test split of wikitext-2 @xcite . for computing the pseudo-perplexity, we first sentencize each text in the dataset, using spacy @xcite . initialization method in preliminary experiments with bert, using random initialization for the prompt's parameters, we observed, similarly to @xcite , that the prompts learned and the performance depend to a large extent on the initialization. we also examined for each based on these preliminary findings, we evaluated the performance of 4 different types of initialization methods, using a prompt length of 3 tokens in each case: words used for each type of initialization are listed in tables 9 , 10 . we sentencize each text before computing the pseudoperplexities. results are averaged across all trials, with a 95%conf idenceinterval (±) and standard deviation (in parentheses). debias. the pseudo-perplexities are listed in in case of bert, we remark that among the different initialization methods, the female biased initialization yields the best results both in terms of debiasing and retaining of language modeling performance. between random, gendered balanced and neutral initialization, the results are similar overall. compared to other debiasing techniques, the female biased initialization is second to inlp in terms of debiasing, while the other initialization types are on par with sentencede-bias. however, our method generally results in a good language modeling score and limited increase in pseudo-perplexity, while inlp significantly reduces the lm score. we suspect this might be due to inlp removing all gender-related information for the model's output. in case of roberta, we notice that the neutral initialization achieves better results than the other initialization types. while this achieves sig-nificantly better results on seat compared to all other debiasing techniques, its results on stereoset are surpassed by both inlp and sentencede-bias. as for bert, results show that our method generally succeeds in maintaining the language modelling ability of the base model. effect on other types of biases besides mitigating the targeted bias and the impact on language modeling performance, the side effects on other biases should also be considered. we evaluate our gender debiased models on the profession, race and religion tests in stereoset and report the stereotype scores in in the profession bias, test there is a significant decrease in bias for all initialization methods, most probably due to the dataset used for training, which is focused on genders and professions. there is no significant effect for race bias. for religion bias, the results vary notably across different trials with the same type of initialization, and there is, on average, an increase in bias for models trained using random initialization. these results suggest extending this method for targeting multiple biases might be feasible. ## conclusions we proposed and investigated a method of reducing social biases in pretrained llms based on prompt tuning, which involves training the prompt embeddings on a small set of templates. in addition to debiasing, this method is also designed to prevent the model from 'forgetting' the identity of the social groups targeted during debiasing. we experimented with this method for reducing gender bias in bert and roberta models. results show that it achieves debiasing performance on par with state-of-the-art while maintaining information about the social group's identity and with a low impact on general language modeling performance. while we focused our experiments on mitigating a single type of social bias in bert and roberta, the method can be adapted to other types of biases and llms.