Detoxifying Language Models Risks Marginalizing Minority Voices (2021)

Abstract: Language models (LMs) must be both safe and equitable to be responsibly deployed in practice. With safety in mind, numerous detoxification techniques (e.g., Dathathri et al. 2020; Krause et al. 2020) have been proposed to mitigate toxic LM generations. In this work, we show that these detoxification techniques hurt equity: they decrease the utility of LMs on language used by marginalized groups (e.g., African-American English and minority identity mentions). In particular, we perform automatic and human evaluations of text generation quality when LMs are conditioned on inputs with different dialects and group identifiers. We find that detoxification makes LMs more brittle to distribution shift, especially on language used by marginalized groups. We identify that these failures stem from detoxification methods exploiting spurious correlations in toxicity datasets. Overall, our results highlight the tension between the controllability and distributional robustness of LMs.

## introduction recent neural language models (lms) have shown enormous improvements in text generation abilities. a key factor behind these improvements is large training corpora that are collected from online sources @xcite . unfortunately, because such corpora are too large to filter granularly @xcite , they inevitably contain so-called toxic examples: undesirable language such as expletives, slurs, or other offensive and threatening speech. when trained on such data, lms inevitably learn to generate toxic text @xcite . to address this issue, recent work has turned towards detoxifying lms: reducing toxic generations without affecting perplexity or generation quality on nontoxic inputs. existing detoxification strategies involve techniques such as finetuning lms on nontoxic data @xcite or incorporating a toxicity discriminator during decoding @xcite . our evaluation of these techniques shows that they are indeed effective at mitigating toxicity, but at what cost? we demonstrate that detoxification can hurt lm utility on language used by minority groups. concretely, we evaluate detoxified lms on text with minority identity mentions (e.g., words such as "gay" or "muslim") and surface markers of african-american english @xcite . we first show that, compared to text containing white-aligned english (wae), detoxification causes a disproportionately large increase in lm perplexity on text with aae and minority identity mentions. moreover, increasing the strength of detoxification amplifies this bias. the same trends hold when evaluating the text generation quality of lms using crowdworkers. when conditioned on wae text, detoxified lms can roughly maintain the topic, fluency, and style of an input prompt. however, generation quality deteriorates when models are conditioned on aae text, i.e., detoxification hurts an lms' ability to understand and complete aae text. we identify that these failures are due to the use of biased toxic classification data. in particular, toxicity datasets often contain spurious correlations between the toxic label and the presence of aae and minority identity mentions @xcite . these correlations cause detoxification techniques to steer generations away from aae and minority identity mentions because they often consider these aspects of language to be toxic. we conclude by outlining concrete harms and possible solutions to these biases. with regard to harms, we argue that biased systems force marginalized users to code-switch or hide their identity and that these systems can contribute to social stigmas. for solutions, we discuss improved procedures for data annotation and model training that may help debias detoxification techniques. ## methods and experimental setup the goal of detoxification is to mitigate the frequency of toxic generations (also called hate speech or offensive language) without affecting an lm's utility or generation quality on nontoxic inputs. we detoxify models using controllable generation techniques that steer outputs away from toxicity. following past work @xcite , we use four techniques that provide state-of-the-art levels of detoxification. ## detoxifying lms introduces biases in this section, we evaluate the detoxification methods and show that they introduce biases into lms that may harm marginalized groups. detoxification topicality fluency style 0% 10% 20% 30% 40% 50% 60% 70% 80% percent preferred over gpt-2 dapt wae dapt aae pplm wae pplm aae gedi wae gedi aae filtering wae filtering aae figure 3: we use the detoxified lms to generate completions of wae or aae prompts. we ask crowdworkers to compare the generations to those from a baseline gpt-2 model. detoxification methods cause a degradation in generation quality (topicality, fluency, and style) when models are conditioned on wae texts. worse yet, generation quality is noticeably worse when conditioned on aae texts, demonstrating unwanted biases. see ## why detoxification introduces biases in this section, we explain why detoxification causes the utility of lms to degrade on text that contains aae and minority identity mentions. first, note that all detoxification techniques make use of labeled toxic/nontoxic data. for example, dapt uses this data directly: it finetunes the lm on nontoxic examples. pplm, gedi, and filtering use this data indirectly: they train a classifier or lm on the toxicity data and then incorporate this model into the lm's decoding strategy. unfortunately, there are spurious correlations between the toxic label and the presence of aae and minority identity mentions @xcite . these correlations arise from annotation and sampling biases. annotation bias occurs because crowdworkers are often unfamiliar with aae and consequently misjudge it as toxic @xcite . sampling bias occurs because many toxic comments are directed towards marginalized groups @xcite . the result of these two biases is that text which contains aae and minority identity mentions is labeled as toxic at disproportionately high rates @xcite . detoxification techniques inherit these undesirable biases. for example, dapt will train lms to not only forget toxicity but also forget aae and minority identity mentions. similarly, the discriminators used by pplm, gedi, and filtering will guide the generated text away from aae and identity mentions because the discriminators typically consider such text as toxic @xcite @xcite . also note that in all of the above cases, increasing the detoxifica-tion strength (e.g., longer finetuning for dapt or higher ω for gedi) exacerbates these problems. in our experiments, we test multiple detoxification methods to show that this bias is not linked to a specific technique, but instead to the process of detoxification in the presence of biased supervised data. in fact, other controllable generation techniques, including prompts @xcite @xcite or conditional lms @xcite will likely exhibit the same type of biases. ## harms of detoxification our results demonstrate that the current state of detoxification poses representational harms @xcite to minority groups. we discuss the concrete impacts of these harms below. in-group harms detoxified lms are deployed in downstream nlp systems in which they directly engage with end users. in addition to lms not being able to generate minority identity mentions and minority dialects, our results suggest that detoxified lms also struggle to understand these aspects of language. this could lead to scenarios where end users who are aae speakers must code-switch to wae to ensure that nlp systems work effectively for them. aside from being an annoyance, this is also a microaggression that poses psychological harms and may discourage aae speakers from engaging with nlp systems whatsoever. stigmatization of language detoxified models also have a propensity to avoid certain topics, e.g., mentioning a minority identity term. as a practical example, the (detoxified) microsoft zo chatbot was capable of discussing christianity but could not discuss islam (stuart-ulin, 2018). failures like these further two types of stigma. first, having one's identity silenced by an nlp system can lead to self-stigmatization and long-term health consequences. second, a lack of informed, conscious discussion on topics of identity or dialect can magnify existing societal stigmas. for example, aligning an lm solely with wae stigmatizes aae as incorrect or "bad" english @xcite . in the technology industry, this can perpetuate a dangerous expectation that aae users are not consumers who matter, stymieing progress on equitable nlp systems. biases are not limited to detoxification although we have focused on problems with detoxification in this paper, similar failures will occur whenever controllable generation methods are used. for example, a common goal is to control the sentiment of generated text @xcite . unfortunately, since sentiment datasets are often biased against certain racial groups @xcite , controlling the sentiment of text will also affect which races are discussed. ## future work: towards bias-free detoxification the harms that we have identified occur largely due to spurious correlations in toxicity datasets. a natural direction for future work is to thus improve datasets, for example, by changing the annotation procedure @xcite or labeling scheme @xcite . unfortunately, this can also make collecting annotations more expensive. as an alternative or in addition to higher quality data, there is growing interest in training accurate models in the presence of biased data @xcite . unfortunately, state-of-the-art debiasing methods are still far from perfect @xcite . we plan to explore new methods for debiasing both datasets and models in future work.