Challenges in Detoxifying Language Models (2021)

Abstract: Large language models (LM) generate remarkably fluent text and can be efficiently adapted across NLP tasks. Measuring and guaranteeing the quality of generated text in terms of safety is imperative for deploying LMs in the real world; to this end, prior work often relies on automatic evaluation of LM toxicity. We critically discuss this approach, evaluate several toxicity mitigation strategies with respect to both automatic and human evaluation, and analyze consequences of toxicity mitigation in terms of model bias and LM quality. We demonstrate that while basic intervention strategies can effectively optimize previously established automatic metrics on the REALTOXICITYPROMPTS dataset, this comes at the cost of reduced LM coverage for both texts about, and dialects of, marginalized groups. Additionally, we find that human raters often disagree with high automatic toxicity scores after strong toxicity reduction interventions—highlighting further the nuances involved in careful evaluation of LM toxicity.

## introduction contemporary text generation models @xcite are capable of generating harmful language, including hate speech, insults, profanities and threats @xcite . these harms are often grouped under the umbrella term "toxicity". 1 to enable safe language model (lm) use and deployment, it is necessary to measure, understand the origins, and undertake effective steps to mitigate toxic text generation in lms. prior work has considered various approaches towards reducing lm toxicity, either by fine-tuning a pre-trained lm @xcite , figure 1 : unintended side effect of automatic toxicity reduction methods: over-filtering of text about marginalized groups reduces the ability of the lm to generate text about these groups, even in a positive way. by steering a model's generation towards text less likely to be classified as toxic @xcite @xcite , or through direct test-time filtering @xcite . recently, gehman et al. (2020) introduced automatic metrics for lm toxicity evaluation based on toxicity scores of the widely used and commercially deployed perspective api model trained on online comments annotated for toxicity. 2 in this paper, we critically discuss both toxicity evaluation and mitigation for contemporary transformer-based english lms. we conduct studies with both human annotation and classifier-based evaluation, to evaluate the effectiveness of different toxicity mitigation methods, and investigate tradeoffs with respect to lm quality and social bias. our contributions are as follows: ## related work while detecting hate speech and offensive language @xcite @xcite , mostly in the context of online community moderation, has long been a subject of research; the study of toxic text generated by language models is a more recent direction. @xcite first demonstrated that synthetic text prompts can cause racist model continuations with gpt-2. gehman et al. (2020) extended the analysis of lm toxicity to non-synthetic prompts, further investigating the effectiveness of multiple potential mitigation approaches. we build on, and extend this work, critically discussing previously introduced metrics to assess lm toxicity, and compare classifier-based lm toxicity scoring with human evaluation. among the most promising approaches for lm toxicity reduction is steering generation towards text less likely to be classified as toxic @xcite . this typically relies on an external toxicity classifier, although @xcite show that even a lm's own toxicity self-diagnosis can be used to this end. toxic language detection systems are known to be biased against specific social groups, and similar to @xcite , we distinguish two bias types. first, classification bias can manifest as topic-related biases, where text mentioning particular identities leads to false positives in toxicity classifiers-e.g. lgbtq+ identity terms ("gay"). this phenomenon has been linked to an increased relative prevalence of identity terms among toxic samples @xcite @xcite . a second type of bias considers disparate performance across dialects, where classifiers on average assign higher toxicity scores e.g. to african-american english (aae) @xcite . a potential side-effect of applying classifier-based toxicity mitigation methods in an lm context, then, is that such biases might also be inherited by the resulting model. our findings are consistent with contemporary work by @xcite demonstrating that lm toxicity mitigations can amplify social biases. our work expands these results across a broader range of models, demographics, and datasets, and uses wikipedia metadata @xcite rather than keyword-matching for measuring topic-related biases. we also show that models which perform well under our and their likelihood-based metrics can still exacerbate bias. finally, by upsampling toxic samples, we can estimate overall lm toxicity, whereas a comparison-based approach can emphasize minor changes to already non-toxic lm completions. other work on toxicity in generated text includes @xcite , who investigate safety specifically in a dialogue setting, and translating existing offensive text into non-offensive variants (nogueira dos @xcite . ## toxic language and lms toxicity following the definition developed by perspective api, we consider an utterance to be toxic if it is rude, disrespectful, or unreasonable language that is likely to make someone leave a discussion. this definition has been adopted by prior work on lm toxicity @xcite , and allows for direct comparability of quantitative results. however, we note two important caveats. first, under this definition, toxicity judgements are subjective, and depend on both the raters evaluating toxicity and their cultural background @xcite , as well as the inferred context. as an example, historical inequalities could lead to a higher toleration of offensive speech among disadvantaged groups, and measurements of toxicity should consider such potential disparities. phenomena where subjective toxicity ratings can differ include sarcasm and utterances of political discontent; we show some example utterances in second, this notion of toxicity only covers one aspect of possible lm harms @xcite . for example, lms can perpetuate harmful stereotypes, or display biases which only manifest statistically over many samples @xcite @xcite . though important, we do not address these here. lm safety criteria are both application-and audience-specific, and in this regard, we recommend caution in over-generalizing results from our work, particularly regarding the absolute and relative efficacy of specific techniques. these caveats are consistent with the limitations our experiments highlight: regarding the relationship between human and automatic toxic evaluation (section 6), and the trade-offs between toxicity mitigation and coverage for marginalized groups (section 8). ## model and methods we next describe the lm we evaluate, as well as three methods we consider for reducing the lm's toxicity, covering both data-based, controllable generation, and direct filtering-based approaches. our standard lm is a transformerxl model @xcite trained on the c4 dataset @xcite , with 24 layers, 16 heads, d @xmath0, and d @xmath1. the model contains 1.4b parameters, and achieves a loss-per-token of 2.40 on the c4 validation set. it uses a 32,000 subword vocabulary with a sentencepiece tokenizer @xcite . we train all lm variants on 128 google cloud tpuv3 cores using the adam optimizer, a batch size of 256 for a total of 3 × 10 5 training steps-about 5 days. for all sampling we use nucleus sampling @xcite , with top-@xmath2.9. ## classifier-based toxicity evaluation although our primary targets are based on human evaluation of lm toxicity, described in section 6, we first describe our evaluation using automatic toxicity metrics for consistency with prior work. we note that several limitations of automated toxicitydetection tools have been well documented, both by jigsaw and by other work @xcite . for automated, classifier-based toxicity evaluation we rely on the realtoxicityprompts (rtp) benchmark @xcite . the aim is to measure lm toxicity within a 20 token continuation, in both the prompt-conditional and unconditional settings. for the conditional case, rtp consists of 100k english web language prompts, with each prompt labelled as either toxic or nontoxic. the rtp metrics are derived from the per-spective api toxicity classifier, which outputs a calibrated toxicity score between 0 and 1. 5 5 it is worth noting that the toxicity scores provided by perspective api are calibrated and intended to reflect the probability of the given text being toxic. that is, text with a score of 0.7 does not indicate that the toxicity level of the sample is more severe than that of text with score 0.5; but instead that the classifier has more certainty in its prediction for the former case, and that for the latter case the model's given these scores, rtp reports two metrics: i) expected maximum toxicity measures the maximum toxicity score given 25 continuations for a given prompt, averaged across prompts; ii) probability of toxicity measures how frequently at least one continuation has a toxicity @xmath3.5, given 25 lm-generated continuations per prompt. ## evaluation via human annotation following the previous section on automated lm toxicity evaluation, we will next measure toxicity and lm generation quality using human evaluation. ## consequences on lm quality to understand consequences of applying lm toxicity interventions, and their potential impact on text generation, we next consider their effect on lm loss, text sample quality, and lm toxicity prediction ability. effect on language modeling loss effect on lm's ability to detect toxicity when training on a toxicity-filtered lm corpus (threshold 0.05), we notice a modest drop in the f 1score (to 0.73; -0.05 points) of the pplm toxicity classifier, which is trained on the lm's representations. this could potentially negatively impact self-debiasing strategies @xcite . ## social bias amplification fairness with respect to all identity groups is crucial if lms are to be used in the real world. two properties, that we highlight as necessary (but insufficient) for fairness are that lms should both be able to model text about topics related to different identity groups (i.e. topic coverage), and also text by people from different identity groups and with different dialects (i.e. dialect coverage). previous works have shown that toxicity classifiers often show lower performance for text written by, or referring to marginalized identity groups @xcite . given that many detoxification techniques heavily rely on toxicity classifiers, we investigate how detoxification affects topic and dialect coverage with respect to different identity groups. we also discuss potential representational harms @xcite which can arise from disparities in the effectiveness of lm toxicity mitigation across different dialects. ## conclusion in this work, we have examined and discussed challenges of lm toxicity evaluation and side-effects of automatic toxicity mitigation using a combination of relatively simple toxicity reduction approaches and previously published methods. we have highlighted the discrepancy between conventional metrics of toxicity and what is perceived by humans. this points towards a research roadmap of defining metrics that better align with perceived toxicity, defining sub-types of toxicity, and including sep-arate test sets for each sub-type. we have further identified a transfer of toxicity classifier bias onto lms, which supports the importance of debiasing toxicity classifiers. based on our results, we additionally highlight the following challenges in mitigating toxic language in lms. first, toxicity is subjective and context dependent -what is considered toxic may differ across cultures, social groups, and personal experiences. though existing methods can effectively optimize automatic toxicity scores, precisely defining what we should measure is an open challenge. ultimately, this will be dependent on users and applications, and requires cross-disciplinary expertise and input from a broad variety of groups. secondly, very low automatic toxicity metrics of state-of-the-art lms after application of the evaluated mitigation techniques suggest that further improvement with respect to these metrics is limited. it is unclear if further optimization against automatic toxicity metrics will lead to improvements in toxicity as judged by humans, or only intensify unintended and problematic side effects of automatic detoxification. we also point out limitations in collecting human ratings, including potential negative psychological impact on annotators. finally, our detoxification increases lm loss, and introduces and amplifies social biases in topic and dialect coverage, potentially leading to decreased lm performance for marginalized groups. we note that although this problem exists in current methods, this tradeoff is not necessarily unavoidable, particularly if future work enables less biased classifiers. alongside toxicity, future work should consider other metrics, such as loss gaps for different topics and dialects. as noted in section 8.3, loss gaps are an imperfect metric; future work on developing quantitative metrics for lm bias could help better understand trade-offs in mitigating toxicity.