Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021)

Abstract: This paper contains prompts and model outputs that are offensive in nature. When trained on large, unfiltered crawls from the Internet, language models pick up and reproduce all kinds of undesirable biases that can be found in the data: They often generate racist, sexist, violent, or otherwise toxic language. As large models require millions of training examples to achieve good performance, it is difficult to completely prevent them from being exposed to such content. In this paper, we first demonstrate a surprising finding: Pretrained language models recognize, to a considerable degree, their undesirable biases and the toxicity of the content they produce. We refer to this capability as self-diagnosis. Based on this finding, we then propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text. We refer to this approach as self-debiasing. Self-debiasing does not rely on manually curated word lists, nor does it require any training data or changes to the model’s parameters. While we by no means eliminate the issue of language models generating biased text, we believe our approach to be an important step in this direction.1

## introduction pretraining neural networks using a language modeling objective leads to large improvements across a variety of natural language processing tasks @xcite @xcite . with model sizes continually increasing @xcite @xcite , ever-larger pretraining datasets are necessary both to prevent overfitting and to provide access to as much world knowledge as possible. however, such large datasets are typically based on crawls from the internet that are only filtered with some basic rules @xcite . as a consequence, they contain non-negligible amounts of text exhibiting biases that are undesirable or outright harmful for many potential applications @xcite . unsurprisingly, language models trained on such data pick up, reproduce, or even amplify these biases @xcite @xcite . simple solutions such as using a list of banned words @xcite ) fall short of mitigating this problem for at least two reasons. first, they do not reliably keep language models from generating biased text: examples in figure 1 show that biased text can easily be generated by using only words that are, by themselves, completely unproblematic. as many such words are important words of the english vocabulary and thus needed for meaningful text generation, they should not be included in a list of banned words. secondly, banning words also prevents language models from gaining knowledge of topics related to the banned words, which may be necessary for some applications. 2 it is therefore inherently difficult to ban words without doing harm to a model's capabilities. building training datasets with more care and deliberation, an alternative solution discussed by @xcite , is important, especially for improving linguistic and cultural diversity in online and other forms of communication. however, for large language models that are available for common global languages, it is desirable to also have other mechanisms to address bias because figure 1 : most probable according to t5-xl @xcite and gpt2-xl @xcite as well as their self-debiased (sd) variants for four different . read ''t5+sd( )'' as: the t5-xl model self-debiased against racism. see §4 for details of the debiasing method. in this paper, we therefore propose an approach that, instead of trusting that a model will implicitly learn desired behaviors from the training data, makes explicit how we expect it to behave at test time: if the model is told which biases are undesired-and it is able to discern their presence-it should be able to avoid them even if they are present in some of the texts it has been trained on. as it is a necessary condition for this approach, we first explore whether language models are able to detect when their own outputs exhibit undesirable attributes, based only on their internal knowledge-a process to which we refer as self-diagnosis. we then investigate whether this ability can be used to perform self-debiasing, that is, whether language models can use this knowledge to discard undesired behaviors in a fully unsupervised fashion. to this end, we propose a decoding algorithm that reduces the probability of a model producing biased text, requiring nothing more than a textual description of the undesired behavior, which can be as simple as a single keyword (e.g., ''sexist @xcite in figure 1 ; see §4 for details). while our results demonstrate that large models in particular are, to some extent, capable of performing self-diagnosis and self-debiasing, we also find that their current capabilities are by no means sufficient to eliminate the issue of corpus-based bias in nlp. ## related work there is a large body of work illustrating that both static (e.g., @xcite and contextualized word embeddings (e.g., @xcite pretrained in a self-supervised fashion exhibit all kinds of unfair and discriminative biases @xcite @xcite @xcite @xcite @xcite and are prone to generating toxic texts @xcite @xcite . for static word embeddings, various algorithms for debiasing have been proposed @xcite @xcite , many of them being based on predefined word lists or other external resources. @xcite propose using dictionary definitions for debiasing, eliminating the need for predefined word lists. for contextualized embeddings, similar methods to alleviate the issue of undesirable biases and toxicity have been proposed @xcite @xcite @xcite . for text generation, gehman et al. (2020) propose domain-adaptive pretraining on non-toxic corpora as outlined by @xcite and consider plug and play language models @xcite . in contrast to our proposed approach, all of these ideas rely either on large sets of training examples or on external resources such as manually curated word lists. our approach for performing self-diagnosis builds heavily on recent work that explores zeroshot learning using task descriptions @xcite @xcite . our proposed self-debiasing algorithm bears some resemblance with prefixconstrained decoding used in interactive machine translation for completing partial translations @xcite . it is also similar to prompt-or keyword-based approaches for controllable text generation @xcite @xcite but these approaches (i) require either a customized pretraining objective or labeled training data, and (ii) use natural language prompts to inform a language model about the task to be solved or the topic to generate text about; in contrast, our target is to control the language model's behavior irrespective of its actual task. ## self-diagnosis we investigate the capability of pretrained language models to detect when their outputs exhibit socially undesirable attributes; we call this process self-diagnosis. 3 in particular, we are interested in whether language models are able to do so without any training data or other external resources, as large enough datasets do not exist for many relevant biases and behaviors and also do not exist for many languages. that is, we only provide a short textual attribute description and rely entirely on the internal knowledge that the model has acquired during pretraining. given a language model m and a sequence of tokens w 1 , . . . , w k , let p m (w | w 1 , . . . , w k ) denote the probability that the language model assigns to w being the next token. for each sentence x generated by m and each attribute description y, we construct a self-diagnosis input sdg(x, y) using the template shown in figure 2 @xcite . that is, we supplement the generated text x with the question whether it contains attribute y and prompt the model to generate an answer to this question. for example, if m generates the text @xmath0, we ask it to provide a continuation for the following input (i.e., a continuation for sdg(x, y), see figure 2(a) ): figure 2 : templates used for self-diagnosis (a) and self-debiasing (b,c) given an input text x and an attribute description or bias type y. see ## self-debiasing in analogy to self-diagnosis, we define selfdebiasing as a language model using only its internal knowledge to adapt its generation process in a way that reduces the probability of generating biased texts. as before, let m be a pretrained language model and y be the textual description of an attribute (see a simple choice for the scaling function would be to set α(x) = 1[@xmath1. through this formulation, changes made to the distribution p m are minimally invasive in that the probability of a word is only altered if this is really deemed necessary; probabilities for words that are not considered biased (i.e., where δ(w, x, y) ≥ 0) are left exactly as is. however, forcing the probability of some words to be exactly zero makes it impossible to compute perplexity for evaluating the quality of a language model, as assigning a probability of zero to the correct next token just once would result in an infinitely large perplexity. instead of forcing the probability of biased words to be zero, we thus resort to a soft variant where their probability is reduced based on the magnitude of the difference δ(w, x, y): so that using word w as a continuation of x is penalized if it has a higher probability according to at least one self-debiasing input. ## conclusion in this paper, we have shown that large language models are capable of performing self-diagnosis, that is, of investigating their own outputs with regards to the presence of undesirable attributes using only their internal knowledge and textual descriptions. based on this finding, we have proposed a decoding algorithm that reduces the probability of a model generating biased text by comparing the original probability of a token with its probability if undesired behavior is explicitly encouraged. as our evaluation is limited to two english datasets covering only a small portion of potentially undesired behaviors in an imperfect fashion, it is important to extend our analysis to other kinds of behaviors and biases, languages, benchmarks, and models. it is clear that self-diagnosis and self-debiasing only reduce and do not eliminate corpus-based bias. for this reason, they are not a viable path towards bias-free models if used in isolation. however, we hope that future work can leverage our proposals, for example, by combining them with complementary models or by extending them to build stronger debiasing solutions.